2024-09-18T13:55:10.142 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-18T13:55:10.161 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T13:55:10.234 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080 branch: wip-bharath11-testing-2024-09-18-1115-quincy description: rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} email: skanta@redhat.com first_in_suite: false job_id: '7911080' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-bharath11-testing-2024-09-18-1115-quincy ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - \(MON_DOWN\) sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath11-testing-2024-09-18-1115-quincy sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 - osd.3 seed: 4631 sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath11-testing-2024-09-18-1115-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 targets: smithi016.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: null - ceph: conf: osd: osd min pg log entries: 5 osd_fast_fail_on_connection_refused: false log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ - \(POOL_APP_NOT_ENABLED\) pre-mgr-commands: - sudo ceph config set mgr mgr_pool false --force - osd_recovery.test_incomplete_pgs: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-18_09:13:00 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-18T13:55:10.234 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa; will attempt to use it 2024-09-18T13:55:10.234 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa/tasks 2024-09-18T13:55:10.235 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-18T13:55:10.236 INFO:teuthology.task.internal:Checking packages... 2024-09-18T13:55:10.258 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '7061ea6ca50908d104e9f3979dc615e118d9e153' 2024-09-18T13:55:10.258 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-18T13:55:10.258 INFO:teuthology.packaging:ref: None 2024-09-18T13:55:10.258 INFO:teuthology.packaging:tag: None 2024-09-18T13:55:10.258 INFO:teuthology.packaging:branch: wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T13:55:10.258 INFO:teuthology.packaging:sha1: 7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T13:55:10.258 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-bharath11-testing-2024-09-18-1115-quincy 2024-09-18T13:55:10.407 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1645-g7061ea6c-1focal 2024-09-18T13:55:10.410 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-18T13:55:10.416 INFO:teuthology.task.internal:no buildpackages task found 2024-09-18T13:55:10.416 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-18T13:55:10.537 INFO:teuthology.task.internal:Saving configuration 2024-09-18T13:55:10.549 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-18T13:55:10.555 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-18T13:55:10.575 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi016.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-18 13:49:15.679194', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-18T13:55:10.575 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-18T13:55:10.581 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2024-09-18T13:55:10.581 INFO:teuthology.run_tasks:Running task console_log... 2024-09-18T13:55:10.613 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fee31bf7ac0>, signals=[15]) 2024-09-18T13:55:10.613 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-18T13:55:10.619 INFO:teuthology.task.internal:Opening connections... 2024-09-18T13:55:10.619 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T13:55:10.621 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:55:10.725 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-18T13:55:10.731 DEBUG:teuthology.orchestra.run.smithi016:> uname -m 2024-09-18T13:55:10.851 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2024-09-18T13:55:10.851 DEBUG:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2024-09-18T13:55:10.899 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="Ubuntu" 2024-09-18T13:55:10.899 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:ID=ubuntu 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE=debian 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="20.04" 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_CODENAME=focal 2024-09-18T13:55:10.900 INFO:teuthology.orchestra.run.smithi016.stdout:UBUNTU_CODENAME=focal 2024-09-18T13:55:10.901 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2024-09-18T13:55:10.920 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-18T13:55:10.928 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-18T13:55:10.939 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-18T13:55:10.939 DEBUG:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2024-09-18T13:55:10.944 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-18T13:55:10.950 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-18T13:55:10.950 DEBUG:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2024-09-18T13:55:11.001 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-18T13:55:11.027 INFO:teuthology.run_tasks:Running task kernel... 2024-09-18T13:55:11.043 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-18T13:55:11.043 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-18T13:55:11.043 DEBUG:teuthology.orchestra.run.smithi016:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-18T13:55:11.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T13:55:11.049 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-18T13:55:11.055 INFO:teuthology.orchestra.run.smithi016.stdout:5.4.0-124-generic 2024-09-18T13:55:11.056 INFO:teuthology.task.kernel:Running kernel on smithi016: 5.4.0-124-generic 2024-09-18T13:55:11.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get clean 2024-09-18T13:55:11.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2024-09-18T13:55:11.395 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T13:55:11.396 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-18T13:55:11.396 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-18T13:55:11.460 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-18T13:55:11.778 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [817 kB] 2024-09-18T13:55:11.863 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,223 kB] 2024-09-18T13:55:11.905 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [475 kB] 2024-09-18T13:55:11.908 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-18T13:55:11.909 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-18T13:55:11.909 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,152 kB] 2024-09-18T13:55:11.917 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,036 kB] 2024-09-18T13:55:11.935 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [441 kB] 2024-09-18T13:55:11.938 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T13:55:11.939 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-18T13:55:11.944 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,012 kB] 2024-09-18T13:55:11.951 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-18T13:55:11.953 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.4 kB] 2024-09-18T13:55:11.953 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-18T13:55:11.953 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-18T13:55:11.953 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-18T13:55:11.957 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-18T13:55:11.987 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,597 kB] 2024-09-18T13:55:12.026 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [553 kB] 2024-09-18T13:55:12.030 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-18T13:55:12.030 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-18T13:55:12.031 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,279 kB] 2024-09-18T13:55:12.051 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [459 kB] 2024-09-18T13:55:12.054 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-18T13:55:12.055 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-18T13:55:12.060 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,231 kB] 2024-09-18T13:55:12.067 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-18T13:55:12.070 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.3 kB] 2024-09-18T13:55:12.070 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-18T13:55:12.070 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-18T13:55:12.070 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-18T13:55:12.078 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-18T13:55:12.079 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-18T13:55:12.079 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-18T13:55:12.079 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-18T13:55:12.095 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-18T13:55:12.096 INFO:teuthology.orchestra.run.smithi016.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-18T13:55:12.096 INFO:teuthology.orchestra.run.smithi016.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-18T13:55:16.416 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 22.0 MB in 3s (6,987 kB/s) 2024-09-18T13:55:17.996 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-18T13:55:18.024 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T13:55:18.101 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-18T13:55:18.327 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2024-09-18T13:55:18.328 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2024-09-18T13:55:18.547 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2024-09-18T13:55:18.547 INFO:teuthology.orchestra.run.smithi016.stdout: linux-generic linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T13:55:18.547 INFO:teuthology.orchestra.run.smithi016.stdout: linux-headers-generic linux-image-5.4.0-196-generic 2024-09-18T13:55:18.547 INFO:teuthology.orchestra.run.smithi016.stdout: linux-modules-5.4.0-196-generic linux-modules-extra-5.4.0-196-generic 2024-09-18T13:55:18.548 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2024-09-18T13:55:18.548 INFO:teuthology.orchestra.run.smithi016.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-18T13:55:18.548 INFO:teuthology.orchestra.run.smithi016.stdout:Recommended packages: 2024-09-18T13:55:18.549 INFO:teuthology.orchestra.run.smithi016.stdout: thermald 2024-09-18T13:55:18.612 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2024-09-18T13:55:18.612 INFO:teuthology.orchestra.run.smithi016.stdout: linux-headers-5.4.0-196 linux-headers-5.4.0-196-generic 2024-09-18T13:55:18.612 INFO:teuthology.orchestra.run.smithi016.stdout: linux-image-5.4.0-196-generic linux-modules-5.4.0-196-generic 2024-09-18T13:55:18.612 INFO:teuthology.orchestra.run.smithi016.stdout: linux-modules-extra-5.4.0-196-generic 2024-09-18T13:55:18.614 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages will be upgraded: 2024-09-18T13:55:18.614 INFO:teuthology.orchestra.run.smithi016.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-18T13:55:18.797 INFO:teuthology.orchestra.run.smithi016.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T13:55:18.797 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 77.2 MB of archives. 2024-09-18T13:55:18.797 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-18T13:55:18.797 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-196-generic amd64 5.4.0-196.216 [15.0 MB] 2024-09-18T13:55:19.934 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-196-generic amd64 5.4.0-196.216 [10.5 MB] 2024-09-18T13:55:20.205 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-196-generic amd64 5.4.0-196.216 [39.2 MB] 2024-09-18T13:55:21.582 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.196.194 [1,900 B] 2024-09-18T13:55:21.582 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.196.194 [2,592 B] 2024-09-18T13:55:21.582 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196 all 5.4.0-196.216 [11.0 MB] 2024-09-18T13:55:21.858 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-196-generic amd64 5.4.0-196.216 [1,386 kB] 2024-09-18T13:55:21.905 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.196.194 [2,460 B] 2024-09-18T13:55:22.606 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 77.2 MB in 3s (23.5 MB/s) 2024-09-18T13:55:22.799 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-modules-5.4.0-196-generic. 2024-09-18T13:55:24.500 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-18T13:55:24.504 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../0-linux-modules-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T13:55:24.621 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:26.496 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-image-5.4.0-196-generic. 2024-09-18T13:55:26.513 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../1-linux-image-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T13:55:26.616 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:27.733 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-196-generic. 2024-09-18T13:55:27.747 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T13:55:27.791 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:32.912 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../3-linux-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T13:55:33.013 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T13:55:33.356 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T13:55:33.500 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-image-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T13:55:33.828 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-headers-5.4.0-196. 2024-09-18T13:55:33.845 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../5-linux-headers-5.4.0-196_5.4.0-196.216_all.deb ... 2024-09-18T13:55:33.886 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T13:55:39.457 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package linux-headers-5.4.0-196-generic. 2024-09-18T13:55:39.478 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../6-linux-headers-5.4.0-196-generic_5.4.0-196.216_amd64.deb ... 2024-09-18T13:55:39.515 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:42.494 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.196.194_amd64.deb ... 2024-09-18T13:55:42.615 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking linux-headers-generic (5.4.0.196.194) over (5.4.0.124.125) ... 2024-09-18T13:55:42.960 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-5.4.0-196 (5.4.0-196.216) ... 2024-09-18T13:55:43.095 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:43.246 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-headers-generic (5.4.0.196.194) ... 2024-09-18T13:55:43.372 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-modules-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:45.968 INFO:teuthology.orchestra.run.smithi016.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-196-generic 2024-09-18T13:55:45.968 INFO:teuthology.orchestra.run.smithi016.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-196-generic 2024-09-18T13:55:46.118 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-modules-extra-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:47.953 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-image-generic (5.4.0.196.194) ... 2024-09-18T13:55:48.063 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up linux-generic (5.4.0.196.194) ... 2024-09-18T13:55:48.181 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for linux-image-5.4.0-196-generic (5.4.0-196.216) ... 2024-09-18T13:55:48.250 INFO:teuthology.orchestra.run.smithi016.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-18T13:55:48.250 INFO:teuthology.orchestra.run.smithi016.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-196-generic 2024-09-18T13:56:11.779 INFO:teuthology.orchestra.run.smithi016.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-18T13:56:11.780 INFO:teuthology.orchestra.run.smithi016.stdout:Sourcing file `/etc/default/grub' 2024-09-18T13:56:11.799 INFO:teuthology.orchestra.run.smithi016.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T13:56:11.818 INFO:teuthology.orchestra.run.smithi016.stdout:Generating grub configuration file ... 2024-09-18T13:56:12.148 INFO:teuthology.orchestra.run.smithi016.stdout:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T13:56:12.171 INFO:teuthology.orchestra.run.smithi016.stdout:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T13:56:12.562 INFO:teuthology.orchestra.run.smithi016.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T13:56:12.574 INFO:teuthology.orchestra.run.smithi016.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T13:56:15.081 INFO:teuthology.orchestra.run.smithi016.stdout:done 2024-09-18T13:56:15.562 DEBUG:teuthology.orchestra.run.smithi016:> dpkg -s linux-image-generic 2024-09-18T13:56:15.620 INFO:teuthology.orchestra.run.smithi016.stdout:Package: linux-image-generic 2024-09-18T13:56:15.620 INFO:teuthology.orchestra.run.smithi016.stdout:Status: install ok installed 2024-09-18T13:56:15.620 INFO:teuthology.orchestra.run.smithi016.stdout:Priority: optional 2024-09-18T13:56:15.620 INFO:teuthology.orchestra.run.smithi016.stdout:Section: kernel 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Installed-Size: 20 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: amd64 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Source: linux-meta 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Version: 5.4.0.196.194 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Recommends: thermald 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:Description: Generic Linux kernel image 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: This package will always depend on the latest generic kernel image 2024-09-18T13:56:15.621 INFO:teuthology.orchestra.run.smithi016.stdout: available. 2024-09-18T13:56:15.621 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-196-generic 2024-09-18T13:56:15.622 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-18T13:56:15.622 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-18T13:56:15.622 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-18T13:56:15.622 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2024-09-18T13:56:15.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get clean 2024-09-18T13:56:15.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2024-09-18T13:56:15.980 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T13:56:15.986 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T13:56:16.068 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T13:56:16.150 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T13:56:17.897 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-18T13:56:17.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-18T13:56:17.997 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-18T13:56:18.198 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2024-09-18T13:56:18.198 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2024-09-18T13:56:18.440 INFO:teuthology.orchestra.run.smithi016.stdout:linux-image-generic is already the newest version (5.4.0.196.194). 2024-09-18T13:56:18.440 INFO:teuthology.orchestra.run.smithi016.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-18T13:56:18.441 DEBUG:teuthology.orchestra.run.smithi016:> dpkg -s linux-image-generic 2024-09-18T13:56:18.462 INFO:teuthology.orchestra.run.smithi016.stdout:Package: linux-image-generic 2024-09-18T13:56:18.462 INFO:teuthology.orchestra.run.smithi016.stdout:Status: install ok installed 2024-09-18T13:56:18.462 INFO:teuthology.orchestra.run.smithi016.stdout:Priority: optional 2024-09-18T13:56:18.462 INFO:teuthology.orchestra.run.smithi016.stdout:Section: kernel 2024-09-18T13:56:18.462 INFO:teuthology.orchestra.run.smithi016.stdout:Installed-Size: 20 2024-09-18T13:56:18.462 INFO:teuthology.orchestra.run.smithi016.stdout:Maintainer: Ubuntu Kernel Team 2024-09-18T13:56:18.462 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: amd64 2024-09-18T13:56:18.462 INFO:teuthology.orchestra.run.smithi016.stdout:Source: linux-meta 2024-09-18T13:56:18.463 INFO:teuthology.orchestra.run.smithi016.stdout:Version: 5.4.0.196.194 2024-09-18T13:56:18.463 INFO:teuthology.orchestra.run.smithi016.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-18T13:56:18.463 INFO:teuthology.orchestra.run.smithi016.stdout:Depends: linux-image-5.4.0-196-generic, linux-modules-extra-5.4.0-196-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-18T13:56:18.463 INFO:teuthology.orchestra.run.smithi016.stdout:Recommends: thermald 2024-09-18T13:56:18.463 INFO:teuthology.orchestra.run.smithi016.stdout:Description: Generic Linux kernel image 2024-09-18T13:56:18.463 INFO:teuthology.orchestra.run.smithi016.stdout: This package will always depend on the latest generic kernel image 2024-09-18T13:56:18.463 INFO:teuthology.orchestra.run.smithi016.stdout: available. 2024-09-18T13:56:18.463 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-18T13:56:18.512 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.1c02iUSGws 2024-09-18T13:56:18.512 DEBUG:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub/grub.cfg /tmp/tmp.1c02iUSGws 2024-09-18T13:56:18.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.1c02iUSGws 2024-09-18T13:56:18.574 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.1c02iUSGws is 9KB 2024-09-18T13:56:18.589 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.1c02iUSGws 2024-09-18T13:56:18.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-18T13:56:18.653 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T13:56:18.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-18T13:56:18.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-18T13:56:18.682 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-196-generic 2024-09-18T13:56:18.682 DEBUG:teuthology.orchestra.run.smithi016:> sudo update-grub 2024-09-18T13:56:20.014 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub' 2024-09-18T13:56:20.022 INFO:teuthology.orchestra.run.smithi016.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-18T13:56:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2024-09-18T13:56:20.327 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.4.0-196-generic 2024-09-18T13:56:20.339 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.4.0-196-generic 2024-09-18T13:56:20.738 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-18T13:56:20.749 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-18T13:56:22.202 INFO:teuthology.orchestra.run.smithi016.stderr:done 2024-09-18T13:56:22.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2024-09-18T13:56:52.237 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-18T13:56:52.237 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-18T13:56:52.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:57:10.589 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2024-09-18T13:57:19.597 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-18T13:57:19.597 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:58:19.657 DEBUG:teuthology.orchestra.remote:timed out 2024-09-18T13:58:31.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-18T13:58:31.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T13:58:31.925 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-18T13:58:32.595 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-18T13:58:32.595 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-196-generic"... 2024-09-18T13:58:32.596 DEBUG:teuthology.orchestra.run.smithi016:> uname -r 2024-09-18T13:58:32.602 INFO:teuthology.orchestra.run.smithi016.stdout:5.4.0-196-generic 2024-09-18T13:58:32.602 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-196-generic vs 5.4.0-196-generic 2024-09-18T13:58:32.602 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-18T13:58:32.602 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-18T13:58:33.603 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-18T13:58:33.603 DEBUG:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-18T13:58:33.681 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2024-09-18T13:58:33.702 DEBUG:teuthology.parallel:result is None 2024-09-18T13:58:33.702 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-18T13:58:33.709 INFO:teuthology.task.internal:Creating test directory... 2024-09-18T13:58:33.709 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-18T13:58:33.716 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-18T13:58:33.726 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-18T13:58:33.736 INFO:teuthology.task.internal:Creating archive directory... 2024-09-18T13:58:33.736 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-18T13:58:33.752 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-18T13:58:33.758 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-18T13:58:33.758 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-18T13:58:33.809 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:58:33.819 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-18T13:58:33.821 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-18T13:58:33.828 INFO:teuthology.task.internal:Configuring sudo... 2024-09-18T13:58:33.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-18T13:58:33.840 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-18T13:58:33.848 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-18T13:58:33.848 DEBUG:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-18T13:58:33.856 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-18T13:58:33.903 DEBUG:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-18T13:58:33.910 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T13:58:33.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-18T13:58:33.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2024-09-18T13:58:33.965 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-18T13:58:33.975 INFO:teuthology.task.internal:Starting timer... 2024-09-18T13:58:33.976 INFO:teuthology.run_tasks:Running task pcp... 2024-09-18T13:58:33.989 INFO:teuthology.run_tasks:Running task selinux... 2024-09-18T13:58:33.997 DEBUG:teuthology.task.selinux:Excluding smithi016: OS 'ubuntu' does not support SELinux 2024-09-18T13:58:33.997 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-18T13:58:33.997 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-18T13:58:33.997 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-18T13:58:33.997 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-18T13:58:34.005 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-18T13:58:34.017 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-18T13:58:34.020 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi016.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-18T14:04:08.269 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi016.front.sepia.ceph.com')] 2024-09-18T14:04:08.271 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-18T14:04:08.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi016.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-18T14:04:08.363 DEBUG:teuthology.orchestra.run.smithi016:> true 2024-09-18T14:04:08.490 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi016.front.sepia.ceph.com' 2024-09-18T14:04:08.491 INFO:teuthology.run_tasks:Running task clock... 2024-09-18T14:04:08.502 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-18T14:04:08.502 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-18T14:04:08.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T14:04:08.535 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-18T14:04:08.535 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: Command line: ntpd -gq 2024-09-18T14:04:08.536 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: proto: precision = 0.242 usec (-22) 2024-09-18T14:04:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T14:04:08.537 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-18T14:04:08.537 INFO:teuthology.orchestra.run.smithi016.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-18T14:04:08.538 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: restrict ::: KOD does nothing without LIMITED. 2024-09-18T14:04:08.538 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: Listen and drop on 0 v6wildcard [::]:123 2024-09-18T14:04:08.538 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-18T14:04:08.539 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: Listen normally on 2 lo 127.0.0.1:123 2024-09-18T14:04:08.539 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: Listen normally on 3 ens1f0 172.21.15.16:123 2024-09-18T14:04:08.539 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: Listen normally on 4 lo [::1]:123 2024-09-18T14:04:08.539 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1564%4]:123 2024-09-18T14:04:08.539 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:08 ntpd[13626]: Listening on routing socket on fd #22 for interface updates 2024-09-18T14:04:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:09 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:09 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:09 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:09 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:09.536 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:09 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:09.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:09 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:11.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:11.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:11.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:11 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:13.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:13.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:13 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:15.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:15.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:15.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:15.537 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/rawstats.20240918: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-18T14:04:15.538 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/peerstats.20240918: Permission denied 2024-09-18T14:04:15.539 INFO:teuthology.orchestra.run.smithi016.stdout:18 Sep 14:04:15 ntpd[13626]: ntpd: time slew +0.002378 s 2024-09-18T14:04:15.539 INFO:teuthology.orchestra.run.smithi016.stdout:ntpd: time slew +0.002378s 2024-09-18T14:04:15.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-18T14:04:15.539 INFO:teuthology.orchestra.run.smithi016.stderr:18 Sep 14:04:15 ntpd[13626]: can't open /var/log/ntpstats/loopstats.20240918: Permission denied 2024-09-18T14:04:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T14:04:15.601 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2024-09-18T14:04:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:04:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:04:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:04:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-18T14:04:15.602 INFO:teuthology.run_tasks:Running task install... 2024-09-18T14:04:15.615 DEBUG:teuthology.task.install:project ceph 2024-09-18T14:04:15.615 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}} 2024-09-18T14:04:15.616 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'} 2024-09-18T14:04:15.616 INFO:teuthology.task.install:Using flavor: default 2024-09-18T14:04:15.621 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-18T14:04:15.621 INFO:teuthology.task.install:extra packages: [] 2024-09-18T14:04:15.621 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-key list | grep Ceph 2024-09-18T14:04:15.725 INFO:teuthology.orchestra.run.smithi016.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-18T14:04:15.785 INFO:teuthology.orchestra.run.smithi016.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-18T14:04:15.786 INFO:teuthology.orchestra.run.smithi016.stdout:uid [ unknown] Ceph.com (release key) 2024-09-18T14:04:15.787 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-18T14:04:15.787 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:04:15.928 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default/ 2024-09-18T14:04:15.928 INFO:teuthology.task.install.deb:Package version is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:04:16.040 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:04:16.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-18T14:04:16.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo apt-get update 2024-09-18T14:04:16.198 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-18T14:04:16.198 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-18T14:04:16.204 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-18T14:04:16.221 INFO:teuthology.orchestra.run.smithi016.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-18T14:04:16.253 INFO:teuthology.orchestra.run.smithi016.stdout:Ign:5 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal InRelease 2024-09-18T14:04:16.275 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-18T14:04:16.297 INFO:teuthology.orchestra.run.smithi016.stdout:Ign:7 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal Release.gpg 2024-09-18T14:04:16.319 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main i386 Packages [3,390 B] 2024-09-18T14:04:16.354 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 Packages [17.4 kB] 2024-09-18T14:04:16.882 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 28.4 kB in 1s (43.1 kB/s) 2024-09-18T14:04:18.085 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-18T14:04:18.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=17.2.7-1645-g7061ea6c-1focal cephadm=17.2.7-1645-g7061ea6c-1focal ceph-mds=17.2.7-1645-g7061ea6c-1focal ceph-mgr=17.2.7-1645-g7061ea6c-1focal ceph-common=17.2.7-1645-g7061ea6c-1focal ceph-fuse=17.2.7-1645-g7061ea6c-1focal ceph-test=17.2.7-1645-g7061ea6c-1focal ceph-volume=17.2.7-1645-g7061ea6c-1focal radosgw=17.2.7-1645-g7061ea6c-1focal python3-rados=17.2.7-1645-g7061ea6c-1focal python3-rgw=17.2.7-1645-g7061ea6c-1focal python3-cephfs=17.2.7-1645-g7061ea6c-1focal python3-rbd=17.2.7-1645-g7061ea6c-1focal libcephfs2=17.2.7-1645-g7061ea6c-1focal libcephfs-dev=17.2.7-1645-g7061ea6c-1focal librados2=17.2.7-1645-g7061ea6c-1focal librbd1=17.2.7-1645-g7061ea6c-1focal rbd-fuse=17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:04:18.183 INFO:teuthology.orchestra.run.smithi016.stdout:Reading package lists... 2024-09-18T14:04:18.406 INFO:teuthology.orchestra.run.smithi016.stdout:Building dependency tree... 2024-09-18T14:04:18.406 INFO:teuthology.orchestra.run.smithi016.stdout:Reading state information... 2024-09-18T14:04:18.621 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages were automatically installed and are no longer required: 2024-09-18T14:04:18.622 INFO:teuthology.orchestra.run.smithi016.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-18T14:04:18.622 INFO:teuthology.orchestra.run.smithi016.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-18T14:04:18.622 INFO:teuthology.orchestra.run.smithi016.stdout:The following additional packages will be installed: 2024-09-18T14:04:18.623 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-18T14:04:18.623 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-18T14:04:18.623 INFO:teuthology.orchestra.run.smithi016.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-18T14:04:18.623 INFO:teuthology.orchestra.run.smithi016.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-18T14:04:18.623 INFO:teuthology.orchestra.run.smithi016.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-18T14:04:18.623 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python-natsort-doc python-pastedeploy-tpl python3-atomicwrites python3-attr 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse python3-ceph-common python3-cherrypy3 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyasn1-modules python3-pyinotify python3-pytest python3-repoze.lru 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-requests-oauthlib python3-routes python3-rsa python3-scipy 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-18T14:04:18.624 INFO:teuthology.orchestra.run.smithi016.stdout: python3-sklearn python3-sklearn-lib python3-soupsieve python3-tempita 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: python3-waitress python3-wcwidth python3-webencodings python3-webob 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: python3-websocket python3-webtest python3-werkzeug qttranslations5-l10n 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: socat xmlstarlet 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout:Suggested packages: 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-18T14:04:18.625 INFO:teuthology.orchestra.run.smithi016.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-18T14:04:18.626 INFO:teuthology.orchestra.run.smithi016.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-18T14:04:18.626 INFO:teuthology.orchestra.run.smithi016.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-18T14:04:18.626 INFO:teuthology.orchestra.run.smithi016.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-18T14:04:18.626 INFO:teuthology.orchestra.run.smithi016.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-18T14:04:18.626 INFO:teuthology.orchestra.run.smithi016.stdout:Recommended packages: 2024-09-18T14:04:18.626 INFO:teuthology.orchestra.run.smithi016.stdout: btrfs-tools 2024-09-18T14:04:18.752 INFO:teuthology.orchestra.run.smithi016.stdout:The following NEW packages will be installed: 2024-09-18T14:04:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-18T14:04:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-18T14:04:18.752 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-18T14:04:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: fonts-mathjax jq libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 2024-09-18T14:04:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: libjs-mathjax libjs-sphinxdoc libjs-underscore liblbfgsb0 liblttng-ust0 2024-09-18T14:04:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: liblua5.3-0 liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-18T14:04:18.753 INFO:teuthology.orchestra.run.smithi016.stdout: libqt5network5 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli python-natsort-doc 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python-pastedeploy-tpl python3-atomicwrites python3-attr python3-bcrypt 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cryptography 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-mako python3-markupsafe python3-natsort python3-oauthlib 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-openssl python3-packaging python3-paste python3-pastedeploy 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pastescript python3-pecan python3-pluggy python3-prettytable 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-repoze.lru python3-requests-oauthlib python3-rgw python3-routes 2024-09-18T14:04:18.754 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2024-09-18T14:04:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-18T14:04:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-18T14:04:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-18T14:04:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-18T14:04:18.755 INFO:teuthology.orchestra.run.smithi016.stdout:The following packages will be upgraded: 2024-09-18T14:04:18.755 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 librbd1 2024-09-18T14:04:18.821 INFO:teuthology.orchestra.run.smithi016.stdout:2 upgraded, 108 newly installed, 0 to remove and 295 not upgraded. 2024-09-18T14:04:18.821 INFO:teuthology.orchestra.run.smithi016.stdout:Need to get 201 MB of archives. 2024-09-18T14:04:18.821 INFO:teuthology.orchestra.run.smithi016.stdout:After this operation, 938 MB of additional disk space will be used. 2024-09-18T14:04:18.821 INFO:teuthology.orchestra.run.smithi016.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-18T14:04:18.905 INFO:teuthology.orchestra.run.smithi016.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-18T14:04:18.905 INFO:teuthology.orchestra.run.smithi016.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 17.2.7-1645-g7061ea6c-1focal [3,507 kB] 2024-09-18T14:04:18.906 INFO:teuthology.orchestra.run.smithi016.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-18T14:04:18.915 INFO:teuthology.orchestra.run.smithi016.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-18T14:04:18.918 INFO:teuthology.orchestra.run.smithi016.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-18T14:04:18.946 INFO:teuthology.orchestra.run.smithi016.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-18T14:04:18.946 INFO:teuthology.orchestra.run.smithi016.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-18T14:04:18.952 INFO:teuthology.orchestra.run.smithi016.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-18T14:04:18.983 INFO:teuthology.orchestra.run.smithi016.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-18T14:04:18.985 INFO:teuthology.orchestra.run.smithi016.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-18T14:04:18.991 INFO:teuthology.orchestra.run.smithi016.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-18T14:04:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-18T14:04:18.993 INFO:teuthology.orchestra.run.smithi016.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-18T14:04:18.995 INFO:teuthology.orchestra.run.smithi016.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-18T14:04:18.996 INFO:teuthology.orchestra.run.smithi016.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-18T14:04:18.996 INFO:teuthology.orchestra.run.smithi016.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-18T14:04:18.997 INFO:teuthology.orchestra.run.smithi016.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-18T14:04:19.001 INFO:teuthology.orchestra.run.smithi016.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-18T14:04:19.015 INFO:teuthology.orchestra.run.smithi016.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-18T14:04:19.016 INFO:teuthology.orchestra.run.smithi016.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-18T14:04:19.016 INFO:teuthology.orchestra.run.smithi016.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-18T14:04:19.017 INFO:teuthology.orchestra.run.smithi016.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-18T14:04:19.018 INFO:teuthology.orchestra.run.smithi016.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-18T14:04:19.019 INFO:teuthology.orchestra.run.smithi016.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-18T14:04:19.019 INFO:teuthology.orchestra.run.smithi016.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-18T14:04:19.021 INFO:teuthology.orchestra.run.smithi016.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-18T14:04:19.022 INFO:teuthology.orchestra.run.smithi016.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-18T14:04:19.035 INFO:teuthology.orchestra.run.smithi016.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-18T14:04:19.036 INFO:teuthology.orchestra.run.smithi016.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-18T14:04:19.036 INFO:teuthology.orchestra.run.smithi016.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-18T14:04:19.052 INFO:teuthology.orchestra.run.smithi016.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-18T14:04:19.052 INFO:teuthology.orchestra.run.smithi016.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-18T14:04:19.053 INFO:teuthology.orchestra.run.smithi016.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-18T14:04:19.054 INFO:teuthology.orchestra.run.smithi016.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-18T14:04:19.056 INFO:teuthology.orchestra.run.smithi016.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-18T14:04:19.061 INFO:teuthology.orchestra.run.smithi016.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-18T14:04:19.061 INFO:teuthology.orchestra.run.smithi016.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-18T14:04:19.069 INFO:teuthology.orchestra.run.smithi016.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-18T14:04:19.070 INFO:teuthology.orchestra.run.smithi016.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-18T14:04:19.071 INFO:teuthology.orchestra.run.smithi016.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-18T14:04:19.086 INFO:teuthology.orchestra.run.smithi016.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-18T14:04:19.087 INFO:teuthology.orchestra.run.smithi016.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-18T14:04:19.087 INFO:teuthology.orchestra.run.smithi016.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-18T14:04:19.169 INFO:teuthology.orchestra.run.smithi016.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-18T14:04:19.246 INFO:teuthology.orchestra.run.smithi016.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-18T14:04:19.248 INFO:teuthology.orchestra.run.smithi016.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-18T14:04:19.264 INFO:teuthology.orchestra.run.smithi016.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-18T14:04:19.265 INFO:teuthology.orchestra.run.smithi016.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-18T14:04:19.265 INFO:teuthology.orchestra.run.smithi016.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-18T14:04:19.265 INFO:teuthology.orchestra.run.smithi016.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-18T14:04:19.266 INFO:teuthology.orchestra.run.smithi016.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-18T14:04:19.266 INFO:teuthology.orchestra.run.smithi016.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-18T14:04:19.266 INFO:teuthology.orchestra.run.smithi016.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-18T14:04:19.267 INFO:teuthology.orchestra.run.smithi016.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-18T14:04:19.268 INFO:teuthology.orchestra.run.smithi016.stdout:Get:56 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 17.2.7-1645-g7061ea6c-1focal [3,562 kB] 2024-09-18T14:04:19.281 INFO:teuthology.orchestra.run.smithi016.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-18T14:04:19.285 INFO:teuthology.orchestra.run.smithi016.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-18T14:04:19.288 INFO:teuthology.orchestra.run.smithi016.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-18T14:04:19.290 INFO:teuthology.orchestra.run.smithi016.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-18T14:04:19.292 INFO:teuthology.orchestra.run.smithi016.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-18T14:04:19.293 INFO:teuthology.orchestra.run.smithi016.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-18T14:04:19.296 INFO:teuthology.orchestra.run.smithi016.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-18T14:04:19.299 INFO:teuthology.orchestra.run.smithi016.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-18T14:04:19.314 INFO:teuthology.orchestra.run.smithi016.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-18T14:04:19.317 INFO:teuthology.orchestra.run.smithi016.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-18T14:04:19.359 INFO:teuthology.orchestra.run.smithi016.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-18T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-18T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-18T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-18T14:04:19.360 INFO:teuthology.orchestra.run.smithi016.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-18T14:04:19.361 INFO:teuthology.orchestra.run.smithi016.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-18T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-18T14:04:19.373 INFO:teuthology.orchestra.run.smithi016.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-18T14:04:19.374 INFO:teuthology.orchestra.run.smithi016.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-18T14:04:19.376 INFO:teuthology.orchestra.run.smithi016.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-18T14:04:19.393 INFO:teuthology.orchestra.run.smithi016.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-18T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-18T14:04:19.394 INFO:teuthology.orchestra.run.smithi016.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-18T14:04:19.397 INFO:teuthology.orchestra.run.smithi016.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-18T14:04:19.398 INFO:teuthology.orchestra.run.smithi016.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-18T14:04:19.415 INFO:teuthology.orchestra.run.smithi016.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 17.2.7-1645-g7061ea6c-1focal [761 kB] 2024-09-18T14:04:19.437 INFO:teuthology.orchestra.run.smithi016.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 17.2.7-1645-g7061ea6c-1focal [414 kB] 2024-09-18T14:04:19.454 INFO:teuthology.orchestra.run.smithi016.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 17.2.7-1645-g7061ea6c-1focal [22.6 kB] 2024-09-18T14:04:19.455 INFO:teuthology.orchestra.run.smithi016.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 17.2.7-1645-g7061ea6c-1focal [200 kB] 2024-09-18T14:04:19.458 INFO:teuthology.orchestra.run.smithi016.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 17.2.7-1645-g7061ea6c-1focal [49.1 kB] 2024-09-18T14:04:19.459 INFO:teuthology.orchestra.run.smithi016.stdout:Get:87 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 17.2.7-1645-g7061ea6c-1focal [390 kB] 2024-09-18T14:04:19.474 INFO:teuthology.orchestra.run.smithi016.stdout:Get:88 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 17.2.7-1645-g7061ea6c-1focal [5,637 kB] 2024-09-18T14:04:19.623 INFO:teuthology.orchestra.run.smithi016.stdout:Get:89 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 17.2.7-1645-g7061ea6c-1focal [116 kB] 2024-09-18T14:04:19.624 INFO:teuthology.orchestra.run.smithi016.stdout:Get:90 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 17.2.7-1645-g7061ea6c-1focal [479 kB] 2024-09-18T14:04:19.630 INFO:teuthology.orchestra.run.smithi016.stdout:Get:91 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 17.2.7-1645-g7061ea6c-1focal [27.0 MB] 2024-09-18T14:04:20.169 INFO:teuthology.orchestra.run.smithi016.stdout:Get:92 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 17.2.7-1645-g7061ea6c-1focal [5,177 kB] 2024-09-18T14:04:20.214 INFO:teuthology.orchestra.run.smithi016.stdout:Get:93 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 17.2.7-1645-g7061ea6c-1focal [223 kB] 2024-09-18T14:04:20.216 INFO:teuthology.orchestra.run.smithi016.stdout:Get:94 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 17.2.7-1645-g7061ea6c-1focal [157 kB] 2024-09-18T14:04:20.217 INFO:teuthology.orchestra.run.smithi016.stdout:Get:95 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 17.2.7-1645-g7061ea6c-1focal [1,594 kB] 2024-09-18T14:04:20.237 INFO:teuthology.orchestra.run.smithi016.stdout:Get:96 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 17.2.7-1645-g7061ea6c-1focal [4,688 kB] 2024-09-18T14:04:20.315 INFO:teuthology.orchestra.run.smithi016.stdout:Get:97 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 17.2.7-1645-g7061ea6c-1focal [23.1 MB] 2024-09-18T14:04:20.795 INFO:teuthology.orchestra.run.smithi016.stdout:Get:98 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 17.2.7-1645-g7061ea6c-1focal [3,936 B] 2024-09-18T14:04:20.795 INFO:teuthology.orchestra.run.smithi016.stdout:Get:99 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 17.2.7-1645-g7061ea6c-1focal [872 kB] 2024-09-18T14:04:20.809 INFO:teuthology.orchestra.run.smithi016.stdout:Get:100 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 17.2.7-1645-g7061ea6c-1focal [2,416 kB] 2024-09-18T14:04:20.826 INFO:teuthology.orchestra.run.smithi016.stdout:Get:101 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 17.2.7-1645-g7061ea6c-1focal [81.4 kB] 2024-09-18T14:04:20.827 INFO:teuthology.orchestra.run.smithi016.stdout:Get:102 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 17.2.7-1645-g7061ea6c-1focal [120 kB] 2024-09-18T14:04:20.828 INFO:teuthology.orchestra.run.smithi016.stdout:Get:103 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 17.2.7-1645-g7061ea6c-1focal [6,653 kB] 2024-09-18T14:04:20.914 INFO:teuthology.orchestra.run.smithi016.stdout:Get:104 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 17.2.7-1645-g7061ea6c-1focal [7,884 kB] 2024-09-18T14:04:21.044 INFO:teuthology.orchestra.run.smithi016.stdout:Get:105 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 17.2.7-1645-g7061ea6c-1focal [4,056 B] 2024-09-18T14:04:21.045 INFO:teuthology.orchestra.run.smithi016.stdout:Get:106 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 17.2.7-1645-g7061ea6c-1focal [53.4 MB] 2024-09-18T14:04:22.173 INFO:teuthology.orchestra.run.smithi016.stdout:Get:107 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 17.2.7-1645-g7061ea6c-1focal [231 kB] 2024-09-18T14:04:22.178 INFO:teuthology.orchestra.run.smithi016.stdout:Get:108 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 17.2.7-1645-g7061ea6c-1focal [21.9 kB] 2024-09-18T14:04:22.179 INFO:teuthology.orchestra.run.smithi016.stdout:Get:109 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 17.2.7-1645-g7061ea6c-1focal [17.2 MB] 2024-09-18T14:04:22.461 INFO:teuthology.orchestra.run.smithi016.stdout:Get:110 https://2.chacra.ceph.com/r/ceph/wip-bharath11-testing-2024-09-18-1115-quincy/7061ea6ca50908d104e9f3979dc615e118d9e153/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 17.2.7-1645-g7061ea6c-1focal [80.6 kB] 2024-09-18T14:04:23.129 INFO:teuthology.orchestra.run.smithi016.stdout:Fetched 201 MB in 4s (54.4 MB/s) 2024-09-18T14:04:23.308 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-18T14:04:23.365 INFO:teuthology.orchestra.run.smithi016.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-18T14:04:23.369 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../000-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-18T14:04:23.448 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T14:04:23.793 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../001-librbd1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:23.919 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librbd1 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T14:04:24.599 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../002-librados2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:24.765 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librados2 (17.2.7-1645-g7061ea6c-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-18T14:04:25.419 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libcephfs2. 2024-09-18T14:04:25.438 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../003-libcephfs2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:25.477 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:25.796 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rados. 2024-09-18T14:04:25.813 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../004-python3-rados_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:25.854 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:26.224 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-18T14:04:26.240 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../005-python3-ceph-argparse_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:04:26.282 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:26.534 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cephfs. 2024-09-18T14:04:26.551 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../006-python3-cephfs_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:26.592 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:26.895 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-18T14:04:26.912 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../007-python3-ceph-common_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:04:26.953 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:27.248 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-prettytable. 2024-09-18T14:04:27.264 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../008-python3-prettytable_0.7.2-5_all.deb ... 2024-09-18T14:04:27.298 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-18T14:04:27.567 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rbd. 2024-09-18T14:04:27.585 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../009-python3-rbd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:27.626 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:27.978 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-18T14:04:27.995 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../010-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-18T14:04:28.036 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T14:04:28.348 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-18T14:04:28.367 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../011-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-18T14:04:28.406 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T14:04:28.717 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-18T14:04:28.733 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../012-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-18T14:04:28.776 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T14:04:29.145 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-18T14:04:29.161 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../013-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-18T14:04:29.203 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T14:04:29.531 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-18T14:04:29.550 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../014-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-18T14:04:29.590 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T14:04:29.993 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-18T14:04:30.012 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../015-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:04:30.062 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:04:30.604 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-18T14:04:30.620 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../016-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:04:30.662 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:04:30.993 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-18T14:04:31.012 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../017-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-18T14:04:31.051 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:04:31.437 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-18T14:04:31.453 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../018-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-18T14:04:31.496 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T14:04:31.849 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package librgw2. 2024-09-18T14:04:31.865 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../019-librgw2_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:31.907 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:32.651 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rgw. 2024-09-18T14:04:32.668 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../020-python3-rgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:32.701 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:33.046 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liboath0. 2024-09-18T14:04:33.063 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../021-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-18T14:04:33.104 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-18T14:04:33.440 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libradosstriper1. 2024-09-18T14:04:33.456 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../022-libradosstriper1_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:33.498 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:33.894 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-common. 2024-09-18T14:04:33.913 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../023-ceph-common_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:33.943 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:36.499 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-base. 2024-09-18T14:04:36.518 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../024-ceph-base_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:36.567 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:37.496 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cryptography. 2024-09-18T14:04:37.512 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../025-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-18T14:04:37.545 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T14:04:37.916 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-openssl. 2024-09-18T14:04:37.932 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../026-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-18T14:04:37.974 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-18T14:04:38.294 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-packaging. 2024-09-18T14:04:38.311 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../027-python3-packaging_20.3-1_all.deb ... 2024-09-18T14:04:38.344 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-18T14:04:38.588 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-18T14:04:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../028-ceph-mgr-modules-core_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:04:38.646 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:39.052 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-18T14:04:39.069 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../029-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-18T14:04:39.110 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T14:04:39.379 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-18T14:04:39.398 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../030-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-18T14:04:39.438 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-18T14:04:39.774 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-natsort. 2024-09-18T14:04:39.794 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../031-python3-natsort_7.0.1-1_all.deb ... 2024-09-18T14:04:39.833 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-18T14:04:40.144 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jwt. 2024-09-18T14:04:40.161 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../032-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-18T14:04:40.203 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T14:04:40.514 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-logutils. 2024-09-18T14:04:40.531 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../033-python3-logutils_0.3.3-7_all.deb ... 2024-09-18T14:04:40.572 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-18T14:04:40.867 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-18T14:04:40.886 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../034-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-18T14:04:40.925 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-18T14:04:41.237 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-mako. 2024-09-18T14:04:41.253 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../035-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-18T14:04:41.295 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T14:04:41.615 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-18T14:04:41.634 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../036-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-18T14:04:41.673 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-18T14:04:42.001 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-18T14:04:42.017 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../037-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-18T14:04:42.059 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T14:04:42.371 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webob. 2024-09-18T14:04:42.388 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../038-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-18T14:04:42.429 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T14:04:42.748 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-18T14:04:42.765 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../039-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-18T14:04:42.807 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T14:04:43.118 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-bs4. 2024-09-18T14:04:43.135 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../040-python3-bs4_4.8.2-1_all.deb ... 2024-09-18T14:04:43.177 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-18T14:04:43.537 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-waitress. 2024-09-18T14:04:43.553 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../041-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-18T14:04:43.598 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T14:04:43.924 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-tempita. 2024-09-18T14:04:43.940 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6_all.deb ... 2024-09-18T14:04:43.982 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-18T14:04:44.294 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-paste. 2024-09-18T14:04:44.311 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../043-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-18T14:04:44.352 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T14:04:44.672 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-18T14:04:44.690 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-18T14:04:44.730 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T14:04:45.033 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-18T14:04:45.054 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-18T14:04:45.091 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-18T14:04:45.395 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webtest. 2024-09-18T14:04:45.412 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../046-python3-webtest_2.0.32-2_all.deb ... 2024-09-18T14:04:45.445 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-18T14:04:45.748 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pecan. 2024-09-18T14:04:45.764 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-18T14:04:45.806 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T14:04:46.153 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-18T14:04:46.169 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../048-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-18T14:04:46.213 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T14:04:46.552 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-18T14:04:46.568 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../049-libsqlite3-mod-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:46.610 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:46.938 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr. 2024-09-18T14:04:46.956 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../050-ceph-mgr_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:46.988 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:47.549 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mon. 2024-09-18T14:04:47.567 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../051-ceph-mon_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:47.607 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:48.386 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-osd. 2024-09-18T14:04:48.404 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../052-ceph-osd_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:48.453 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:50.614 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph. 2024-09-18T14:04:50.630 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../053-ceph_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:50.672 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:50.976 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-fuse. 2024-09-18T14:04:50.994 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../054-ceph-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:51.026 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:51.446 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mds. 2024-09-18T14:04:51.463 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../055-ceph-mds_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:51.504 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:52.024 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package cephadm. 2024-09-18T14:04:52.041 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../056-cephadm_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:04:52.082 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:52.393 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-jinja2. 2024-09-18T14:04:52.409 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../057-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-18T14:04:52.443 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T14:04:53.538 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-18T14:04:53.558 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../058-ceph-mgr-cephadm_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:04:53.588 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:53.966 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-18T14:04:53.986 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../059-python3-repoze.lru_0.7-2_all.deb ... 2024-09-18T14:04:54.024 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-18T14:04:54.260 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjs-underscore. 2024-09-18T14:04:54.277 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../060-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-18T14:04:54.319 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T14:04:54.588 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-18T14:04:54.604 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../061-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-18T14:04:54.646 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T14:04:54.957 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-routes. 2024-09-18T14:04:54.973 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../062-python3-routes_2.4.1-2_all.deb ... 2024-09-18T14:04:55.007 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-18T14:04:55.310 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-18T14:04:55.330 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../063-ceph-mgr-dashboard_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:04:55.368 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:04:56.649 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-decorator. 2024-09-18T14:04:56.668 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../064-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-18T14:04:56.707 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T14:04:57.010 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-18T14:04:57.026 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../065-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-18T14:04:57.068 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T14:04:57.389 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-scipy. 2024-09-18T14:04:57.407 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../066-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-18T14:04:57.447 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-18T14:04:58.740 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-18T14:04:58.759 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../067-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-18T14:04:58.798 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T14:04:59.318 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-joblib. 2024-09-18T14:04:59.339 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../068-python3-joblib_0.14.0-3_all.deb ... 2024-09-18T14:04:59.377 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-18T14:04:59.755 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-sklearn. 2024-09-18T14:04:59.772 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../069-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-18T14:04:59.813 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T14:05:00.481 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-18T14:05:00.497 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../070-ceph-mgr-diskprediction-local_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:05:00.538 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:01.658 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-cachetools. 2024-09-18T14:05:01.679 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../071-python3-cachetools_4.0.0-1_all.deb ... 2024-09-18T14:05:01.716 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-18T14:05:02.036 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-18T14:05:02.053 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../072-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-18T14:05:02.094 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T14:05:02.406 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-18T14:05:02.426 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../073-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-18T14:05:02.464 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T14:05:02.734 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-rsa. 2024-09-18T14:05:02.752 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../074-python3-rsa_4.0-3_all.deb ... 2024-09-18T14:05:02.792 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-18T14:05:03.078 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-google-auth. 2024-09-18T14:05:03.098 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../075-python3-google-auth_1.5.1-2_all.deb ... 2024-09-18T14:05:03.136 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-18T14:05:03.457 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-blinker. 2024-09-18T14:05:03.475 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../076-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-18T14:05:03.516 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T14:05:03.785 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-18T14:05:03.801 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../077-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-18T14:05:03.835 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T14:05:04.156 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-18T14:05:04.172 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../078-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-18T14:05:04.205 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T14:05:04.533 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-websocket. 2024-09-18T14:05:04.551 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../079-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-18T14:05:04.591 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T14:05:04.903 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-18T14:05:04.920 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../080-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-18T14:05:04.961 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T14:05:05.578 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-18T14:05:05.597 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../081-ceph-mgr-k8sevents_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:05:05.640 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:05.953 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-18T14:05:05.969 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../082-libonig5_6.9.4-1_amd64.deb ... 2024-09-18T14:05:06.011 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-18T14:05:06.347 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-18T14:05:06.367 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../083-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T14:05:06.414 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:05:06.692 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package jq. 2024-09-18T14:05:06.709 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../084-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-18T14:05:06.750 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:05:07.011 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package socat. 2024-09-18T14:05:07.028 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../085-socat_1.7.3.3-2_amd64.deb ... 2024-09-18T14:05:07.069 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-18T14:05:07.391 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package xmlstarlet. 2024-09-18T14:05:07.408 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../086-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-18T14:05:07.449 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-18T14:05:07.796 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-test. 2024-09-18T14:05:07.813 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../087-ceph-test_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:05:07.854 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:12.663 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package ceph-volume. 2024-09-18T14:05:12.683 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../088-ceph-volume_17.2.7-1645-g7061ea6c-1focal_all.deb ... 2024-09-18T14:05:12.721 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:13.052 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-18T14:05:13.070 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../089-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T14:05:13.110 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:05:13.616 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-18T14:05:13.634 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../090-libcephfs-dev_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:05:13.674 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:14.069 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package nvme-cli. 2024-09-18T14:05:14.086 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../091-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-18T14:05:14.145 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T14:05:14.466 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-18T14:05:14.483 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../092-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-18T14:05:14.524 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:05:16.131 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-18T14:05:16.152 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../093-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-18T14:05:16.189 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-18T14:05:16.533 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-18T14:05:16.550 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../094-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-18T14:05:16.591 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T14:05:16.886 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-attr. 2024-09-18T14:05:16.907 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../095-python3-attr_19.3.0-2_all.deb ... 2024-09-18T14:05:16.953 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-18T14:05:17.264 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-webencodings. 2024-09-18T14:05:17.284 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../096-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-18T14:05:17.322 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T14:05:17.642 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-html5lib. 2024-09-18T14:05:17.661 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../097-python3-html5lib_1.0.1-2_all.deb ... 2024-09-18T14:05:17.700 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-18T14:05:18.028 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-18T14:05:18.045 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../098-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-18T14:05:18.078 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T14:05:18.556 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pastescript. 2024-09-18T14:05:18.574 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../099-python3-pastescript_2.0.2-4_all.deb ... 2024-09-18T14:05:18.614 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-18T14:05:18.910 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pluggy. 2024-09-18T14:05:18.927 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../100-python3-pluggy_0.13.0-2_all.deb ... 2024-09-18T14:05:18.968 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-18T14:05:19.263 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-psutil. 2024-09-18T14:05:19.283 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../101-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-18T14:05:19.329 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T14:05:19.649 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-py. 2024-09-18T14:05:19.669 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../102-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-18T14:05:19.707 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T14:05:20.052 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-18T14:05:20.070 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../103-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-18T14:05:20.102 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T14:05:20.413 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-18T14:05:20.430 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../104-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-18T14:05:20.471 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T14:05:20.774 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-pytest. 2024-09-18T14:05:20.795 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../105-python3-pytest_4.6.9-1_all.deb ... 2024-09-18T14:05:20.832 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-18T14:05:21.202 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package python3-simplejson. 2024-09-18T14:05:21.223 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../106-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-18T14:05:21.260 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T14:05:21.522 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-18T14:05:21.542 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../107-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-18T14:05:21.571 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T14:05:22.271 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package radosgw. 2024-09-18T14:05:22.291 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../108-radosgw_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:05:22.320 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:24.174 INFO:teuthology.orchestra.run.smithi016.stdout:Selecting previously unselected package rbd-fuse. 2024-09-18T14:05:24.191 INFO:teuthology.orchestra.run.smithi016.stdout:Preparing to unpack .../109-rbd-fuse_17.2.7-1645-g7061ea6c-1focal_amd64.deb ... 2024-09-18T14:05:24.232 INFO:teuthology.orchestra.run.smithi016.stdout:Unpacking rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:24.555 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-18T14:05:24.871 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-18T14:05:24.997 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-18T14:05:25.123 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-18T14:05:25.324 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:25.889 INFO:teuthology.orchestra.run.smithi016.stdout:Adding system user cephadm....done 2024-09-18T14:05:26.042 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:05:26.168 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-18T14:05:26.528 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-18T14:05:26.813 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-18T14:05:26.930 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-18T14:05:27.056 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-18T14:05:27.400 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-18T14:05:27.837 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-18T14:05:28.121 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-18T14:05:28.414 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-18T14:05:28.715 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-18T14:05:29.025 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-ceph-argparse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:29.326 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-18T14:05:29.477 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-18T14:05:29.778 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-18T14:05:30.079 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-18T14:05:30.188 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-18T14:05:30.573 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-18T14:05:30.891 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-18T14:05:31.175 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-18T14:05:31.544 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-18T14:05:31.853 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-18T14:05:32.204 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-18T14:05:32.513 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-18T14:05:32.823 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-18T14:05:32.949 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-18T14:05:33.074 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-18T14:05:33.376 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-18T14:05:33.778 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-18T14:05:34.154 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-18T14:05:34.514 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-18T14:05:34.757 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:35.108 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:05:35.243 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-18T14:05:35.594 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-18T14:05:35.720 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-18T14:05:35.846 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-18T14:05:36.148 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-18T14:05:36.474 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:05:36.600 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-18T14:05:36.726 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-18T14:05:37.046 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-18T14:05:37.339 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-18T14:05:37.640 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-18T14:05:37.766 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-18T14:05:38.050 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-18T14:05:38.360 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-18T14:05:38.669 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-18T14:05:39.080 INFO:teuthology.orchestra.run.smithi016.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-18T14:05:39.313 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-18T14:05:39.439 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-18T14:05:39.774 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-18T14:05:40.134 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-18T14:05:40.259 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-18T14:05:40.386 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-18T14:05:40.745 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-18T14:05:41.046 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librados2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:41.181 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-18T14:05:42.703 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libsqlite3-mod-ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:42.821 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:05:42.947 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-18T14:05:43.331 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-18T14:05:43.641 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-18T14:05:43.942 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-18T14:05:44.068 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-18T14:05:44.427 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libcephfs2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:44.545 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-18T14:05:44.889 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libradosstriper1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:45.015 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-18T14:05:45.566 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-18T14:05:46.655 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librbd1 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:46.781 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-18T14:05:46.907 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-18T14:05:47.033 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-18T14:05:47.158 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-18T14:05:47.468 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-18T14:05:47.594 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-18T14:05:47.912 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-modules-core (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:48.030 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:48.257 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T14:05:48.257 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-18T14:05:48.731 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up libcephfs-dev (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:48.959 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-18T14:05:49.260 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-18T14:05:49.595 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-18T14:05:49.948 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rados (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:50.074 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up librgw2 (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:50.200 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rbd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:50.326 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up rbd-fuse (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:50.452 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-18T14:05:50.745 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-18T14:05:51.030 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-rgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:51.156 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-cephfs (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:51.282 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-18T14:05:52.235 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-common (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:52.730 INFO:teuthology.orchestra.run.smithi016.stdout:Adding group ceph....done 2024-09-18T14:05:53.106 INFO:teuthology.orchestra.run.smithi016.stdout:Adding system user ceph....done 2024-09-18T14:05:53.266 INFO:teuthology.orchestra.run.smithi016.stdout:Setting system user ceph properties....done 2024-09-18T14:05:53.292 INFO:teuthology.orchestra.run.smithi016.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-18T14:05:53.447 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-18T14:05:53.765 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-18T14:05:54.220 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-test (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:54.354 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up radosgw (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:54.630 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T14:05:54.630 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-18T14:05:55.281 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-base (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:55.666 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-18T14:05:56.159 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mds (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T14:05:56.389 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-18T14:05:56.860 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T14:05:57.094 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-18T14:05:57.536 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-osd (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:58.067 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T14:05:58.067 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-18T14:05:58.505 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-k8sevents (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:58.631 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-diskprediction-local (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:58.774 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mon (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:59.001 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T14:05:59.001 INFO:teuthology.orchestra.run.smithi016.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-18T14:05:59.450 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-cephadm (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:59.592 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:59.718 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-mgr-dashboard (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:59.861 INFO:teuthology.orchestra.run.smithi016.stdout:Setting up ceph-volume (17.2.7-1645-g7061ea6c-1focal) ... 2024-09-18T14:05:59.980 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-18T14:06:00.131 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-18T14:06:00.438 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-18T14:06:03.328 INFO:teuthology.orchestra.run.smithi016.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-18T14:06:03.917 INFO:teuthology.orchestra.run.smithi016.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-18T14:06:03.923 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:03.924 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=7061ea6ca50908d104e9f3979dc615e118d9e153 2024-09-18T14:06:04.068 DEBUG:teuthology.orchestra.run.smithi016:> dpkg-query -W -f '${Version}' ceph 2024-09-18T14:06:04.090 INFO:teuthology.orchestra.run.smithi016.stdout:17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:06:04.091 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1645-g7061ea6c-1focal 2024-09-18T14:06:04.091 INFO:teuthology.task.install:The correct ceph version 17.2.7-1645-g7061ea6c-1focal is installed. 2024-09-18T14:06:04.093 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-18T14:06:04.093 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:04.093 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-18T14:06:04.149 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-18T14:06:04.149 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:04.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/daemon-helper 2024-09-18T14:06:04.160 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-18T14:06:04.173 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-18T14:06:04.173 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:04.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-18T14:06:04.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-18T14:06:04.243 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-18T14:06:04.243 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:04.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/usr/bin/stdin-killer 2024-09-18T14:06:04.256 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-18T14:06:04.266 INFO:teuthology.run_tasks:Running task ceph... 2024-09-18T14:06:04.344 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-18T14:06:04.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2024-09-18T14:06:04.360 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-18T14:06:04.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-18T14:06:04.370 INFO:tasks.ceph:Creating extra log directories... 2024-09-18T14:06:04.370 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-18T14:06:04.393 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-18T14:06:04.393 INFO:tasks.ceph:config {'conf': {'osd': {'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-18T14:06:04.393 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080', 'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'description': 'rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7911080', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'osd.3']], 'seed': 4631, 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath11-testing-2024-09-18-1115-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_7061ea6ca50908d104e9f3979dc615e118d9e153/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'targets': {'smithi016.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(POOL_APP_NOT_ENABLED\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', '\\(MON_DOWN\\)'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr_pool false --force'], 'flavor': 'default', 'fs': 'xfs', 'sha1': '7061ea6ca50908d104e9f3979dc615e118d9e153', 'cluster': 'ceph'}}, {'osd_recovery.test_incomplete_pgs': None}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-18_09:13:00', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-18T14:06:04.394 DEBUG:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-18T14:06:04.401 DEBUG:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2024-09-18T14:06:04.459 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:04.459 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/scratch_devs of=/dev/stdout 2024-09-18T14:06:04.465 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-18T14:06:04.465 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 2024-09-18T14:06:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-18T14:06:04.471 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:06:04.471 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 360 Links: 1 2024-09-18T14:06:04.471 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:06:04.471 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-18 14:02:43.019035392 +0000 2024-09-18T14:06:04.472 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-18 14:02:43.015035484 +0000 2024-09-18T14:06:04.472 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-18 14:02:43.015035484 +0000 2024-09-18T14:06:04.472 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-18T14:06:04.472 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-18T14:06:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-18T14:06:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-18T14:06:04.485 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000317736 s, 1.6 MB/s 2024-09-18T14:06:04.486 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-18T14:06:04.494 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 2024-09-18T14:06:04.544 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-18T14:06:04.544 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:06:04.544 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 365 Links: 1 2024-09-18T14:06:04.544 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:06:04.544 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-18 14:02:43.647020910 +0000 2024-09-18T14:06:04.544 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-18 14:02:43.431025890 +0000 2024-09-18T14:06:04.544 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-18 14:02:43.431025890 +0000 2024-09-18T14:06:04.545 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-18T14:06:04.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-18T14:06:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-18T14:06:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-18T14:06:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000293394 s, 1.7 MB/s 2024-09-18T14:06:04.558 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-18T14:06:04.565 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 2024-09-18T14:06:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-18T14:06:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:06:04.616 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 371 Links: 1 2024-09-18T14:06:04.616 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:06:04.616 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-18 14:02:44.091010670 +0000 2024-09-18T14:06:04.616 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-18 14:02:43.871015743 +0000 2024-09-18T14:06:04.616 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-18 14:02:43.871015743 +0000 2024-09-18T14:06:04.616 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-18T14:06:04.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-18T14:06:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-18T14:06:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-18T14:06:04.628 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000293713 s, 1.7 MB/s 2024-09-18T14:06:04.629 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-18T14:06:04.636 DEBUG:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 2024-09-18T14:06:04.687 INFO:teuthology.orchestra.run.smithi016.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-18T14:06:04.687 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-18T14:06:04.688 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 6h/6d Inode: 379 Links: 1 2024-09-18T14:06:04.688 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-18T14:06:04.688 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2024-09-18 14:02:44.491001446 +0000 2024-09-18T14:06:04.688 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2024-09-18 14:02:44.291006058 +0000 2024-09-18T14:06:04.688 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2024-09-18 14:02:44.291006058 +0000 2024-09-18T14:06:04.688 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2024-09-18T14:06:04.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-18T14:06:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2024-09-18T14:06:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2024-09-18T14:06:04.699 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes copied, 0.000288033 s, 1.8 MB/s 2024-09-18T14:06:04.700 DEBUG:teuthology.orchestra.run.smithi016:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-18T14:06:04.707 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T14:06:04.707 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-18T14:06:04.707 INFO:tasks.ceph:Generating config... 2024-09-18T14:06:04.708 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] osd_fast_fail_on_connection_refused = False 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-18T14:06:04.709 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-18T14:06:04.710 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-18T14:06:04.710 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-18T14:06:04.710 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-18T14:06:04.710 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-18T14:06:04.710 INFO:tasks.ceph:[global] ms type = async 2024-09-18T14:06:04.710 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-18T14:06:04.711 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-18T14:06:04.711 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-18T14:06:04.711 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-18T14:06:04.711 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-18T14:06:04.711 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-18T14:06:04.711 INFO:tasks.ceph:Setting up mon.a... 2024-09-18T14:06:04.711 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-18T14:06:04.789 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2024-09-18T14:06:04.792 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-18T14:06:04.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T14:06:04.841 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.b', '[v2:172.21.15.16:3301,v1:172.21.15.16:6790]'), ('mon.c', '[v2:172.21.15.16:3302,v1:172.21.15.16:6791]')] 2024-09-18T14:06:04.842 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],[v2:172.21.15.16:3302,v1:172.21.15.16:6791]', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd min pg log entries': 5, 'osd_fast_fail_on_connection_refused': False, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2024-09-18T14:06:04.842 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:04.843 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T14:06:04.847 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.16 --addv b '[v2:172.21.15.16:3301,v1:172.21.15.16:6790]' --addv c '[v2:172.21.15.16:3302,v1:172.21.15.16:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:06:04.928 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid 946c9667-3cd7-48a9-8c9c-fd6b4af766c9 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:setting min_mon_release = octopus 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 946c9667-3cd7-48a9-8c9c-fd6b4af766c9 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2024-09-18T14:06:04.927768+0000 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-18T14:06:04.927768+0000 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 15 (octopus) 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:election_strategy: 1 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] mon.b 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.16:3302/0,v1:172.21.15.16:6791/0] mon.c 2024-09-18T14:06:04.929 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-18T14:06:04.931 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-18T14:06:04.937 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 946c9667-3cd7-48a9-8c9c-fd6b4af766c9... 2024-09-18T14:06:04.938 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-18T14:06:04.972 INFO:teuthology.orchestra.run.smithi016.stdout:[global] 2024-09-18T14:06:04.972 INFO:teuthology.orchestra.run.smithi016.stdout: chdir = "" 2024-09-18T14:06:04.972 INFO:teuthology.orchestra.run.smithi016.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-18T14:06:04.972 INFO:teuthology.orchestra.run.smithi016.stdout: auth supported = cephx 2024-09-18T14:06:04.972 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: filestore xattr use omap = true 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: mon clock drift allowed = 1.000 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: osd crush chooseleaf type = 0 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: auth debug = true 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on old message = true 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: ms die on bug = true 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: mon pg warn max object skew = 0 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.973 INFO:teuthology.orchestra.run.smithi016.stdout: # disable pg_autoscaler by default for new pools 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default size = 2 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow primary affinity = true 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd allow pg remap = true 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on legacy crush tunables = false 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on crush straw calc version zero = false 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on no sortbitwise = false 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on osd down out interval zero = false 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon warn on too few osds = false 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-18T14:06:04.974 INFO:teuthology.orchestra.run.smithi016.stdout: mon_allow_pool_size_one = true 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: osd default data pool replay window = 5 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: mon allow pool delete = true 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: mon cluster log file level = debug 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: debug asserts on shutdown = true 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: mon health detail to clog = false 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: mon host = "172.21.15.16,[v2:172.21.15.16:3301,v1:172.21.15.16:6790],[v2:172.21.15.16:3302,v1:172.21.15.16:6791]" 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: mon client directed command retry = 5 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: mon election default strategy = 1 2024-09-18T14:06:04.975 INFO:teuthology.orchestra.run.smithi016.stdout: ms inject socket failures = 5000 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: ms type = async 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: fsid = 946c9667-3cd7-48a9-8c9c-fd6b4af766c9 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout:[osd] 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: osd journal size = 100 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub load threshold = 5.0 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: osd scrub max interval = 600 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock profile = high_recovery_ops 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: osd recover clone overlap = true 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: osd recovery max chunk = 1048576 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.976 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug shutdown = true 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug op order = true 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify stray on activate = true 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: osd open classes on start = true 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug pg log writeout = true 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: osd deep scrub update digest min age = 30 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: osd map max advance = 10 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: journal zero on create = true 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: filestore ondisk finisher threads = 3 2024-09-18T14:06:04.977 INFO:teuthology.orchestra.run.smithi016.stdout: filestore apply finisher threads = 3 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: bdev debug aio = true 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug misdirected ops = true 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: osd min pg log entries = 5 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: osd_fast_fail_on_connection_refused = False 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: bdev async discard = True 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: bdev enable discard = True 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore block size = 96636764160 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore fsck on mount = True 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: bluestore zero block detection = True 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluefs = 20 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: debug bluestore = 20 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: debug osd = 20 2024-09-18T14:06:04.978 INFO:teuthology.orchestra.run.smithi016.stdout: debug rocksdb = 10 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd backfillfull_ratio = 0.85 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd full ratio = 0.9 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd nearfull ratio = 0.8 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify cached snaps = True 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: osd debug verify missing on start = True 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: osd failsafe full ratio = 0.95 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: osd mclock override recovery settings = True 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: osd memory target = 1610612736 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: osd objectstore = bluestore 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: osd op queue = debug_random 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: osd op queue cut off = debug_random 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr] 2024-09-18T14:06:04.979 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: debug mgr = 20 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: mgr/telemetry/nag = false 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout:[mon] 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: debug ms = 1 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: debug mon = 20 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: debug paxos = 20 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: debug auth = 20 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: mon data avail warn = 5 2024-09-18T14:06:04.980 INFO:teuthology.orchestra.run.smithi016.stdout: mon mgr mkfs grace = 240 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min pgs per osd = 4 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd reporter subtree level = osd 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: mon osd prime pg temp = true 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: mon reweight min bytes per osd = 10 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: auth mon ticket ttl = 660 # 11m 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: auth service ticket ttl = 240 # 4m 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: # don't complain about insecure global_id in the test suite 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-18T14:06:04.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: # 1m isn't quite enough 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: mon_down_mkfs_grace = 2m 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: mon_warn_on_filestore_osds = false 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: mon scrub interval = 300 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout:[client] 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: rgw cache enabled = true 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable ops log = true 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: rgw enable usage log = true 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.a] 2024-09-18T14:06:04.982 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.b] 2024-09-18T14:06:04.983 INFO:teuthology.orchestra.run.smithi016.stdout:[mon.c] 2024-09-18T14:06:04.983 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-18T14:06:04.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-18T14:06:05.072 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-18T14:06:05.072 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:05.072 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-18T14:06:05.079 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:05.079 DEBUG:teuthology.orchestra.run.smithi016:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-18T14:06:05.084 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:06:05.084 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:05.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-18T14:06:05.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-18T14:06:05.106 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:05.106 DEBUG:teuthology.orchestra.run.smithi016:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:06:05.111 INFO:tasks.ceph:Setting up mon nodes... 2024-09-18T14:06:05.111 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-18T14:06:05.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T14:06:05.198 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-18T14:06:05.201 INFO:tasks.ceph:Setting up mds nodes... 2024-09-18T14:06:05.201 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-18T14:06:05.201 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-18T14:06:05.202 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}} 2024-09-18T14:06:05.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-18T14:06:05.217 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T14:06:05.217 INFO:tasks.ceph:role: osd.0 2024-09-18T14:06:05.217 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:06:05.217 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-18T14:06:05.258 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:06:05.258 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:06:05.258 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:06:05.259 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 2024-09-18T14:06:05.259 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:06:05.259 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:06:05.259 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:06:05.259 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:06:05.259 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:06:05.259 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:06:05.260 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-18T14:06:05.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-18T14:06:05.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-18T14:06:05.334 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:06:05.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:06:05.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-18T14:06:05.346 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T14:06:05.346 INFO:tasks.ceph:role: osd.1 2024-09-18T14:06:05.346 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:06:05.346 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-18T14:06:05.384 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:06:05.384 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:06:05.384 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:06:05.384 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 2024-09-18T14:06:05.385 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:06:05.385 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:06:05.385 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:06:05.385 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:06:05.385 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:06:05.385 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:06:05.386 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-18T14:06:05.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-18T14:06:05.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-18T14:06:05.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:06:05.413 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:06:05.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-18T14:06:05.423 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T14:06:05.424 INFO:tasks.ceph:role: osd.2 2024-09-18T14:06:05.424 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:06:05.424 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-18T14:06:05.463 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:06:05.463 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:06:05.463 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:06:05.463 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 2024-09-18T14:06:05.463 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:06:05.463 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:06:05.464 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:06:05.464 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:06:05.464 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:06:05.464 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:06:05.465 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-18T14:06:05.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-18T14:06:05.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-18T14:06:05.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:06:05.540 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:06:05.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-18T14:06:05.552 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-18T14:06:05.552 INFO:tasks.ceph:role: osd.3 2024-09-18T14:06:05.552 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:06:05.552 DEBUG:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-18T14:06:05.594 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-18T14:06:05.594 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-18T14:06:05.594 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-18T14:06:05.594 INFO:teuthology.orchestra.run.smithi016.stdout: = reflink=1 2024-09-18T14:06:05.594 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-18T14:06:05.594 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2024-09-18T14:06:05.595 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-18T14:06:05.595 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-18T14:06:05.595 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-18T14:06:05.595 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-18T14:06:05.596 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2024-09-18T14:06:05.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-18T14:06:05.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-18T14:06:05.666 INFO:teuthology.orchestra.run.smithi016.stderr:sudo: /sbin/restorecon: command not found 2024-09-18T14:06:05.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:06:05.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:06:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:05.728+0000 7f07ccee5540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-18T14:06:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:05.728+0000 7f07ccee5540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-18T14:06:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:05.728+0000 7f07ccee5540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T14:06:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:05.728+0000 7f07ccee5540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T14:06:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:05.728+0000 7f07ccee5540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-18T14:06:05.734 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:05.728+0000 7f07ccee5540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-18T14:06:06.296 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:06.288+0000 7f07ccee5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:07.097 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:07.092+0000 7f07ccee5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:07.896 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:07.888+0000 7f07ccee5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:08.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-18T14:06:08.758 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:06:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:08.836+0000 7fb1062ee540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-18T14:06:08.843 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:08.836+0000 7fb1062ee540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-18T14:06:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:08.836+0000 7fb1062ee540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T14:06:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:08.836+0000 7fb1062ee540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T14:06:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:08.836+0000 7fb1062ee540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-18T14:06:08.844 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:08.836+0000 7fb1062ee540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-18T14:06:09.396 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:09.388+0000 7fb1062ee540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:10.189 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:10.184+0000 7fb1062ee540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:10.970 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:10.964+0000 7fb1062ee540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:11.813 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-18T14:06:11.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:11.856+0000 7faa0f7cb540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-18T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:11.856+0000 7faa0f7cb540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-18T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:11.856+0000 7faa0f7cb540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T14:06:11.864 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:11.856+0000 7faa0f7cb540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:11.856+0000 7faa0f7cb540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-18T14:06:11.865 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:11.856+0000 7faa0f7cb540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-18T14:06:12.418 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:12.412+0000 7faa0f7cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:13.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:13.188+0000 7faa0f7cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:13.980 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:13.972+0000 7faa0f7cb540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:14.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-18T14:06:14.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:06:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:14.896+0000 7fc26d078540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-18T14:06:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:14.900+0000 7fc26d078540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-18T14:06:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:14.900+0000 7fc26d078540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T14:06:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:14.900+0000 7fc26d078540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T14:06:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:14.900+0000 7fc26d078540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-18T14:06:14.906 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:14.900+0000 7fc26d078540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-18T14:06:15.465 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:15.460+0000 7fc26d078540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:16.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:16.236+0000 7fc26d078540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:17.029 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:06:17.020+0000 7fc26d078540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2024-09-18T14:06:17.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-18T14:06:17.909 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-18T14:06:17.909 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:17.909 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-18T14:06:17.971 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:17.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-18T14:06:17.987 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:17.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-18T14:06:18.000 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:18.000 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-18T14:06:18.009 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:18.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-18T14:06:18.069 INFO:tasks.ceph:Adding keys to all mons... 2024-09-18T14:06:18.070 DEBUG:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-18T14:06:18.078 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2024-09-18T14:06:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBN3upm7pnMCxAAwhbs5zykjy+EXVToqIYIbA== 2024-09-18T14:06:18.078 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2024-09-18T14:06:18.078 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBN3upmylenKxAA0MYHhtOlWj6uZbdcPyaZAw== 2024-09-18T14:06:18.078 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2024-09-18T14:06:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBQ3upmKWY0MhAA3BHqS8GdutaNLUE7OxVAfg== 2024-09-18T14:06:18.079 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2024-09-18T14:06:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBT3upmJo9vMxAAI89K89ZPlxOesj7+wN+Ixg== 2024-09-18T14:06:18.079 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2024-09-18T14:06:18.079 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBW3upmvVbmNRAA9NMUTZq+XbasbLH8RtmqAQ== 2024-09-18T14:06:18.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-18T14:06:18.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:06:18.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:06:18.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:06:18.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-18T14:06:18.344 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-18T14:06:18.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-18T14:06:18.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T14:06:18.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-18T14:06:18.470 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-18T14:06:18.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T14:06:18.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-18T14:06:18.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-18T14:06:18.610 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-18T14:06:18.706 DEBUG:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-18T14:06:18.720 DEBUG:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-18T14:06:18.727 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-18T14:06:18.727 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-18T14:06:18.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-18T14:06:18.771 INFO:tasks.ceph.mon.a:Started 2024-09-18T14:06:18.771 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-18T14:06:18.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-18T14:06:18.773 INFO:tasks.ceph.mon.b:Started 2024-09-18T14:06:18.773 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-18T14:06:18.773 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-18T14:06:18.775 INFO:tasks.ceph.mon.c:Started 2024-09-18T14:06:18.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph config set mgr mgr_pool false --force 2024-09-18T14:06:29.399 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-18T14:06:29.400 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-18T14:06:29.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-18T14:06:29.402 INFO:tasks.ceph.mgr.x:Started 2024-09-18T14:06:29.402 INFO:tasks.ceph:Setting crush tunables to default 2024-09-18T14:06:29.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2024-09-18T14:06:29.592 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2024-09-18T14:06:29.602 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-18T14:06:29.602 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:29.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-18T14:06:29.612 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:29.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-18T14:06:29.623 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:29.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-18T14:06:29.637 DEBUG:teuthology.orchestra.run.smithi016:> set -ex 2024-09-18T14:06:29.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-18T14:06:29.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 08356eed-f55f-427f-bce6-f522a976a2c9 0 2024-09-18T14:06:29.669 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:29.664+0000 7f40a84f5000 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-18T14:06:29.879 INFO:teuthology.orchestra.run.smithi016.stdout:0 2024-09-18T14:06:29.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 851cad7b-f44e-48a3-9d7a-2498617c734e 1 2024-09-18T14:06:30.086 INFO:teuthology.orchestra.run.smithi016.stdout:1 2024-09-18T14:06:30.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 8dc73a5a-05ad-471a-8d5b-442d5605ab45 2 2024-09-18T14:06:30.300 INFO:teuthology.orchestra.run.smithi016.stdout:2 2024-09-18T14:06:30.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 536c2bb9-0d34-4dc1-b8b2-f2df92543320 3 2024-09-18T14:06:30.421 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:30.416+0000 7f40a84f5000 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-18T14:06:30.479 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:30.476+0000 7f40a84f5000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-18T14:06:30.504 INFO:teuthology.orchestra.run.smithi016.stdout:3 2024-09-18T14:06:30.522 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-18T14:06:30.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-18T14:06:30.524 INFO:tasks.ceph.osd.0:Started 2024-09-18T14:06:30.524 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-18T14:06:30.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-18T14:06:30.526 INFO:tasks.ceph.osd.1:Started 2024-09-18T14:06:30.526 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T14:06:30.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T14:06:30.529 INFO:tasks.ceph.osd.2:Started 2024-09-18T14:06:30.529 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T14:06:30.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T14:06:30.532 INFO:tasks.ceph.osd.3:Started 2024-09-18T14:06:30.532 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:06:30.628 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:30.624+0000 7f40a84f5000 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-18T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:30.713 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":6,"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","created":"2024-09-18T14:06:29.125568+0000","modified":"2024-09-18T14:06:30.498654+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T14:06:30.498654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"08356eed-f55f-427f-bce6-f522a976a2c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"851cad7b-f44e-48a3-9d7a-2498617c734e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8dc73a5a-05ad-471a-8d5b-442d5605ab45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"536c2bb9-0d34-4dc1-b8b2-f2df92543320","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:06:30.725 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-18T14:06:30.725 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-18T14:06:30.886 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:30.884+0000 7f40a84f5000 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-18T14:06:30.895 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:06:30.892+0000 7fa216851540 -1 Falling back to public interface 2024-09-18T14:06:30.927 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-18T14:06:30.924+0000 7ff65fd71540 -1 Falling back to public interface 2024-09-18T14:06:30.931 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:06:30.928+0000 7fbb932f1540 -1 Falling back to public interface 2024-09-18T14:06:30.938 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:30.936+0000 7f40a84f5000 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-18T14:06:31.128 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T14:06:31.129 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:31.128+0000 7f40a84f5000 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-18T14:06:31.144 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-18T14:06:31.140+0000 7fbb7c36c540 -1 Falling back to public interface 2024-09-18T14:06:31.199 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:31.196+0000 7f40a84f5000 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-18T14:06:31.266 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:31.264+0000 7f40a84f5000 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-18T14:06:31.305 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-18T14:06:31.305 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":6,"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","created":"2024-09-18T14:06:29.125568+0000","modified":"2024-09-18T14:06:30.498654+0000","last_up_change":"0.000000","last_in_change":"2024-09-18T14:06:30.498654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"08356eed-f55f-427f-bce6-f522a976a2c9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"851cad7b-f44e-48a3-9d7a-2498617c734e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"8dc73a5a-05ad-471a-8d5b-442d5605ab45","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"536c2bb9-0d34-4dc1-b8b2-f2df92543320","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:06:31.316 DEBUG:teuthology.misc:0 of 4 OSDs are up 2024-09-18T14:06:31.693 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:31.692+0000 7f40a84f5000 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-18T14:06:31.818 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:31.816+0000 7f40a84f5000 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-18T14:06:32.147 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:32.144+0000 7f40a84f5000 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-18T14:06:32.313 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:32.308+0000 7f40a84f5000 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-18T14:06:32.380 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:32.376+0000 7f40a84f5000 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-18T14:06:32.434 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:32.432+0000 7f40a84f5000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-18T14:06:32.488 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:32.484+0000 7f40a84f5000 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-18T14:06:32.668 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:32.664+0000 7f40a84f5000 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-18T14:06:32.831 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:32.828+0000 7f40a84f5000 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-18T14:06:32.898 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:32.896+0000 7f40a84f5000 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-18T14:06:33.065 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:33.060+0000 7f40a84f5000 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-18T14:06:33.131 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:33.128+0000 7f40a84f5000 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-18T14:06:33.867 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:33.864+0000 7f40a84f5000 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-18T14:06:34.534 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-18T14:06:34.532+0000 7ff65fd71540 -1 osd.0 0 log_to_monitors true 2024-09-18T14:06:34.720 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:06:34.716+0000 7fa216851540 -1 osd.2 0 log_to_monitors true 2024-09-18T14:06:34.838 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:06:34.836+0000 7fbb932f1540 -1 osd.3 0 log_to_monitors true 2024-09-18T14:06:34.909 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:34.908+0000 7f409156c700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-18T14:06:34.910 INFO:tasks.ceph.mgr.x.smithi016.stderr:2024-09-18T14:06:34.908+0000 7f409156c700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-18T14:06:34.932 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-18T14:06:34.928+0000 7fbb7c36c540 -1 osd.1 0 log_to_monitors true 2024-09-18T14:06:35.967 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-18T14:06:35.964+0000 7ff65cd36700 -1 osd.0 0 waiting for initial osdmap 2024-09-18T14:06:35.968 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:06:35.964+0000 7fbb8ea91700 -1 osd.3 0 waiting for initial osdmap 2024-09-18T14:06:35.969 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:06:35.964+0000 7fbb902b6700 -1 osd.3 0 waiting for initial osdmap 2024-09-18T14:06:35.969 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:06:35.964+0000 7fa213816700 -1 osd.2 0 waiting for initial osdmap 2024-09-18T14:06:35.969 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:06:35.964+0000 7fa213816700 -1 osd.2 0 waiting for initial osdmap 2024-09-18T14:06:35.973 INFO:tasks.ceph.osd.0.smithi016.stderr:2024-09-18T14:06:35.968+0000 7ff656ab9700 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:06:35.974 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:06:35.972+0000 7fbb8a039700 -1 osd.3 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:06:35.974 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:06:35.972+0000 7fa20d599700 -1 osd.2 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:06:35.980 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-18T14:06:35.976+0000 7fbb79331700 -1 osd.1 0 waiting for initial osdmap 2024-09-18T14:06:35.983 INFO:tasks.ceph.osd.1.smithi016.stderr:2024-09-18T14:06:35.980+0000 7fbb728b3700 -1 osd.1 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:06:37.719 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-18T14:06:38.009 INFO:teuthology.misc.health.smithi016.stdout: 2024-09-18T14:06:38.010 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":10,"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","created":"2024-09-18T14:06:29.125568+0000","modified":"2024-09-18T14:06:37.979289+0000","last_up_change":"2024-09-18T14:06:36.969101+0000","last_in_change":"2024-09-18T14:06:30.498654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"08356eed-f55f-427f-bce6-f522a976a2c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":22294},{"type":"v1","addr":"172.21.15.16:6809","nonce":22294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":22294},{"type":"v1","addr":"172.21.15.16:6811","nonce":22294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":22294},{"type":"v1","addr":"172.21.15.16:6815","nonce":22294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":22294},{"type":"v1","addr":"172.21.15.16:6813","nonce":22294}]},"public_addr":"172.21.15.16:6809/22294","cluster_addr":"172.21.15.16:6811/22294","heartbeat_back_addr":"172.21.15.16:6815/22294","heartbeat_front_addr":"172.21.15.16:6813/22294","state":["exists","up"]},{"osd":1,"uuid":"851cad7b-f44e-48a3-9d7a-2498617c734e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":22293},{"type":"v1","addr":"172.21.15.16:6825","nonce":22293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":22293},{"type":"v1","addr":"172.21.15.16:6827","nonce":22293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":22293},{"type":"v1","addr":"172.21.15.16:6831","nonce":22293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":22293},{"type":"v1","addr":"172.21.15.16:6829","nonce":22293}]},"public_addr":"172.21.15.16:6825/22293","cluster_addr":"172.21.15.16:6827/22293","heartbeat_back_addr":"172.21.15.16:6831/22293","heartbeat_front_addr":"172.21.15.16:6829/22293","state":["exists","up"]},{"osd":2,"uuid":"8dc73a5a-05ad-471a-8d5b-442d5605ab45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":22295},{"type":"v1","addr":"172.21.15.16:6801","nonce":22295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":22295},{"type":"v1","addr":"172.21.15.16:6803","nonce":22295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":22295},{"type":"v1","addr":"172.21.15.16:6807","nonce":22295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":22295},{"type":"v1","addr":"172.21.15.16:6805","nonce":22295}]},"public_addr":"172.21.15.16:6801/22295","cluster_addr":"172.21.15.16:6803/22295","heartbeat_back_addr":"172.21.15.16:6807/22295","heartbeat_front_addr":"172.21.15.16:6805/22295","state":["exists","up"]},{"osd":3,"uuid":"536c2bb9-0d34-4dc1-b8b2-f2df92543320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":22310},{"type":"v1","addr":"172.21.15.16:6817","nonce":22310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":22310},{"type":"v1","addr":"172.21.15.16:6819","nonce":22310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":22310},{"type":"v1","addr":"172.21.15.16:6823","nonce":22310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":22310},{"type":"v1","addr":"172.21.15.16:6821","nonce":22310}]},"public_addr":"172.21.15.16:6817/22310","cluster_addr":"172.21.15.16:6819/22310","heartbeat_back_addr":"172.21.15.16:6823/22310","heartbeat_front_addr":"172.21.15.16:6821/22310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.510803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.952904+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.723520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.858647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:06:38.026 DEBUG:teuthology.misc:4 of 4 OSDs are up 2024-09-18T14:06:38.026 INFO:tasks.ceph:Creating RBD pool 2024-09-18T14:06:38.026 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-18T14:06:39.008 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'rbd' created 2024-09-18T14:06:39.019 DEBUG:teuthology.orchestra.run.smithi016:> rbd --cluster ceph pool init rbd 2024-09-18T14:06:42.027 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-18T14:06:42.028 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-18T14:06:42.028 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-18T14:06:42.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-18T14:06:42.029 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-18T14:06:42.359 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:42.376 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":4,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":22116},{"type":"v1","addr":"172.21.15.16:6833","nonce":22116}]},"active_addr":"172.21.15.16:6833/22116","active_change":"2024-09-18T14:06:33.887586+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading asyncssh library:No module named 'asyncssh'","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":3964038091}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":2368845884}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.16:0","nonce":4056027207}]}]} 2024-09-18T14:06:42.378 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-18T14:06:42.378 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-18T14:06:42.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:06:42.667 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:42.667 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","created":"2024-09-18T14:06:29.125568+0000","modified":"2024-09-18T14:06:42.005576+0000","last_up_change":"2024-09-18T14:06:36.969101+0000","last_in_change":"2024-09-18T14:06:30.498654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T14:06:38.348847+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"08356eed-f55f-427f-bce6-f522a976a2c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":22294},{"type":"v1","addr":"172.21.15.16:6809","nonce":22294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":22294},{"type":"v1","addr":"172.21.15.16:6811","nonce":22294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":22294},{"type":"v1","addr":"172.21.15.16:6815","nonce":22294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":22294},{"type":"v1","addr":"172.21.15.16:6813","nonce":22294}]},"public_addr":"172.21.15.16:6809/22294","cluster_addr":"172.21.15.16:6811/22294","heartbeat_back_addr":"172.21.15.16:6815/22294","heartbeat_front_addr":"172.21.15.16:6813/22294","state":["exists","up"]},{"osd":1,"uuid":"851cad7b-f44e-48a3-9d7a-2498617c734e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":22293},{"type":"v1","addr":"172.21.15.16:6825","nonce":22293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":22293},{"type":"v1","addr":"172.21.15.16:6827","nonce":22293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":22293},{"type":"v1","addr":"172.21.15.16:6831","nonce":22293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":22293},{"type":"v1","addr":"172.21.15.16:6829","nonce":22293}]},"public_addr":"172.21.15.16:6825/22293","cluster_addr":"172.21.15.16:6827/22293","heartbeat_back_addr":"172.21.15.16:6831/22293","heartbeat_front_addr":"172.21.15.16:6829/22293","state":["exists","up"]},{"osd":2,"uuid":"8dc73a5a-05ad-471a-8d5b-442d5605ab45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":22295},{"type":"v1","addr":"172.21.15.16:6801","nonce":22295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":22295},{"type":"v1","addr":"172.21.15.16:6803","nonce":22295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":22295},{"type":"v1","addr":"172.21.15.16:6807","nonce":22295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":22295},{"type":"v1","addr":"172.21.15.16:6805","nonce":22295}]},"public_addr":"172.21.15.16:6801/22295","cluster_addr":"172.21.15.16:6803/22295","heartbeat_back_addr":"172.21.15.16:6807/22295","heartbeat_front_addr":"172.21.15.16:6805/22295","state":["exists","up"]},{"osd":3,"uuid":"536c2bb9-0d34-4dc1-b8b2-f2df92543320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":22310},{"type":"v1","addr":"172.21.15.16:6817","nonce":22310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":22310},{"type":"v1","addr":"172.21.15.16:6819","nonce":22310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":22310},{"type":"v1","addr":"172.21.15.16:6823","nonce":22310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":22310},{"type":"v1","addr":"172.21.15.16:6821","nonce":22310}]},"public_addr":"172.21.15.16:6817/22310","cluster_addr":"172.21.15.16:6819/22310","heartbeat_back_addr":"172.21.15.16:6823/22310","heartbeat_front_addr":"172.21.15.16:6821/22310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.510803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.952904+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.723520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.858647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:06:42.681 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-18T14:06:42.681 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:06:42.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:42.966 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","created":"2024-09-18T14:06:29.125568+0000","modified":"2024-09-18T14:06:42.005576+0000","last_up_change":"2024-09-18T14:06:36.969101+0000","last_in_change":"2024-09-18T14:06:30.498654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T14:06:38.348847+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"08356eed-f55f-427f-bce6-f522a976a2c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":22294},{"type":"v1","addr":"172.21.15.16:6809","nonce":22294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":22294},{"type":"v1","addr":"172.21.15.16:6811","nonce":22294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":22294},{"type":"v1","addr":"172.21.15.16:6815","nonce":22294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":22294},{"type":"v1","addr":"172.21.15.16:6813","nonce":22294}]},"public_addr":"172.21.15.16:6809/22294","cluster_addr":"172.21.15.16:6811/22294","heartbeat_back_addr":"172.21.15.16:6815/22294","heartbeat_front_addr":"172.21.15.16:6813/22294","state":["exists","up"]},{"osd":1,"uuid":"851cad7b-f44e-48a3-9d7a-2498617c734e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":22293},{"type":"v1","addr":"172.21.15.16:6825","nonce":22293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":22293},{"type":"v1","addr":"172.21.15.16:6827","nonce":22293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":22293},{"type":"v1","addr":"172.21.15.16:6831","nonce":22293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":22293},{"type":"v1","addr":"172.21.15.16:6829","nonce":22293}]},"public_addr":"172.21.15.16:6825/22293","cluster_addr":"172.21.15.16:6827/22293","heartbeat_back_addr":"172.21.15.16:6831/22293","heartbeat_front_addr":"172.21.15.16:6829/22293","state":["exists","up"]},{"osd":2,"uuid":"8dc73a5a-05ad-471a-8d5b-442d5605ab45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":22295},{"type":"v1","addr":"172.21.15.16:6801","nonce":22295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":22295},{"type":"v1","addr":"172.21.15.16:6803","nonce":22295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":22295},{"type":"v1","addr":"172.21.15.16:6807","nonce":22295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":22295},{"type":"v1","addr":"172.21.15.16:6805","nonce":22295}]},"public_addr":"172.21.15.16:6801/22295","cluster_addr":"172.21.15.16:6803/22295","heartbeat_back_addr":"172.21.15.16:6807/22295","heartbeat_front_addr":"172.21.15.16:6805/22295","state":["exists","up"]},{"osd":3,"uuid":"536c2bb9-0d34-4dc1-b8b2-f2df92543320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":22310},{"type":"v1","addr":"172.21.15.16:6817","nonce":22310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":22310},{"type":"v1","addr":"172.21.15.16:6819","nonce":22310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":22310},{"type":"v1","addr":"172.21.15.16:6823","nonce":22310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":22310},{"type":"v1","addr":"172.21.15.16:6821","nonce":22310}]},"public_addr":"172.21.15.16:6817/22310","cluster_addr":"172.21.15.16:6819/22310","heartbeat_back_addr":"172.21.15.16:6823/22310","heartbeat_front_addr":"172.21.15.16:6821/22310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.510803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.952904+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.723520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.858647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:06:42.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T14:06:42.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T14:06:42.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T14:06:42.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T14:06:43.124 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:43.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:06:43.131 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:43.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:06:43.137 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:43.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:06:43.152 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:43.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:06:43.413 INFO:teuthology.orchestra.run.smithi016.stdout:38654705666 2024-09-18T14:06:43.425 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705666 for osd.1 2024-09-18T14:06:43.427 INFO:teuthology.orchestra.run.smithi016.stdout:38654705666 2024-09-18T14:06:43.439 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705666 for osd.3 2024-09-18T14:06:43.444 INFO:teuthology.orchestra.run.smithi016.stdout:38654705666 2024-09-18T14:06:43.454 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705666 for osd.0 2024-09-18T14:06:43.501 INFO:teuthology.orchestra.run.smithi016.stdout:38654705666 2024-09-18T14:06:43.515 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705666 for osd.2 2024-09-18T14:06:44.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:06:44.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:06:44.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:06:44.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:06:44.747 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:44.760 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.1 2024-09-18T14:06:44.760 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:44.788 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:44.797 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:44.801 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.0 2024-09-18T14:06:44.801 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:44.810 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.3 2024-09-18T14:06:44.811 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:44.817 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:44.830 INFO:tasks.ceph.ceph_manager.ceph:need seq 38654705668 got 38654705668 for osd.2 2024-09-18T14:06:44.831 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:44.831 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T14:06:44.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:45.173 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:45.183 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-18T14:06:43.903594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189912,"kb_used_data":476,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297448,"statfs":{"total":386547056640,"available":385328586752,"internally_reserved":0,"allocated":487424,"data_stored":133232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.889357"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021583+0000","last_change":"2024-09-18T14:06:42.021844+0000","last_active":"2024-09-18T14:06:42.021583+0000","last_peered":"2024-09-18T14:06:42.021583+0000","last_clean":"2024-09-18T14:06:42.021583+0000","last_became_active":"2024-09-18T14:06:40.014703+0000","last_became_peered":"2024-09-18T14:06:40.014703+0000","last_unstale":"2024-09-18T14:06:42.021583+0000","last_undegraded":"2024-09-18T14:06:42.021583+0000","last_fullsized":"2024-09-18T14:06:42.021583+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:46:17.562922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048751900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.829521+0000","last_change":"2024-09-18T14:06:42.829873+0000","last_active":"2024-09-18T14:06:42.829521+0000","last_peered":"2024-09-18T14:06:42.829521+0000","last_clean":"2024-09-18T14:06:42.829521+0000","last_became_active":"2024-09-18T14:06:40.014302+0000","last_became_peered":"2024-09-18T14:06:40.014302+0000","last_unstale":"2024-09-18T14:06:42.829521+0000","last_undegraded":"2024-09-18T14:06:42.829521+0000","last_fullsized":"2024-09-18T14:06:42.829521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:51:59.683672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.016348+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:42.016348+0000","last_peered":"2024-09-18T14:06:42.016348+0000","last_clean":"2024-09-18T14:06:42.016348+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:42.016348+0000","last_undegraded":"2024-09-18T14:06:42.016348+0000","last_fullsized":"2024-09-18T14:06:42.016348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.828995+0000","last_change":"2024-09-18T14:06:42.829393+0000","last_active":"2024-09-18T14:06:42.828995+0000","last_peered":"2024-09-18T14:06:42.828995+0000","last_clean":"2024-09-18T14:06:42.828995+0000","last_became_active":"2024-09-18T14:06:40.014431+0000","last_became_peered":"2024-09-18T14:06:40.014431+0000","last_unstale":"2024-09-18T14:06:42.828995+0000","last_undegraded":"2024-09-18T14:06:42.828995+0000","last_fullsized":"2024-09-18T14:06:42.828995+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:27:29.592710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"12'1","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021867+0000","last_change":"2024-09-18T14:06:42.022150+0000","last_active":"2024-09-18T14:06:42.021867+0000","last_peered":"2024-09-18T14:06:42.021867+0000","last_clean":"2024-09-18T14:06:42.021867+0000","last_became_active":"2024-09-18T14:06:40.013037+0000","last_became_peered":"2024-09-18T14:06:40.013037+0000","last_unstale":"2024-09-18T14:06:42.021867+0000","last_undegraded":"2024-09-18T14:06:42.021867+0000","last_fullsized":"2024-09-18T14:06:42.021867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:54:59.496509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070704800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"14'2","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.017768+0000","last_change":"2024-09-18T14:06:42.017768+0000","last_active":"2024-09-18T14:06:42.017768+0000","last_peered":"2024-09-18T14:06:42.017768+0000","last_clean":"2024-09-18T14:06:42.017768+0000","last_became_active":"2024-09-18T14:06:40.015970+0000","last_became_peered":"2024-09-18T14:06:40.015970+0000","last_unstale":"2024-09-18T14:06:42.017768+0000","last_undegraded":"2024-09-18T14:06:42.017768+0000","last_fullsized":"2024-09-18T14:06:42.017768+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:33:09.363728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.829118+0000","last_change":"2024-09-18T14:06:42.829469+0000","last_active":"2024-09-18T14:06:42.829118+0000","last_peered":"2024-09-18T14:06:42.829118+0000","last_clean":"2024-09-18T14:06:42.829118+0000","last_became_active":"2024-09-18T14:06:40.014952+0000","last_became_peered":"2024-09-18T14:06:40.014952+0000","last_unstale":"2024-09-18T14:06:42.829118+0000","last_undegraded":"2024-09-18T14:06:42.829118+0000","last_fullsized":"2024-09-18T14:06:42.829118+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:58:56.331843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021486+0000","last_change":"2024-09-18T14:06:42.021799+0000","last_active":"2024-09-18T14:06:42.021486+0000","last_peered":"2024-09-18T14:06:42.021486+0000","last_clean":"2024-09-18T14:06:42.021486+0000","last_became_active":"2024-09-18T14:06:40.014543+0000","last_became_peered":"2024-09-18T14:06:40.014543+0000","last_unstale":"2024-09-18T14:06:42.021486+0000","last_undegraded":"2024-09-18T14:06:42.021486+0000","last_fullsized":"2024-09-18T14:06:42.021486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:32:23.233193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044037999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297468,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074372,"statfs":{"total":96636764160,"available":96332156928,"internally_reserved":0,"allocated":114688,"data_stored":30746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":34342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":126976,"data_stored":34361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":33783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:45.184 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:45.471 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:45.484 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2024-09-18T14:06:43.903594+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189912,"kb_used_data":476,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297448,"statfs":{"total":386547056640,"available":385328586752,"internally_reserved":0,"allocated":487424,"data_stored":133232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"1.889357"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021583+0000","last_change":"2024-09-18T14:06:42.021844+0000","last_active":"2024-09-18T14:06:42.021583+0000","last_peered":"2024-09-18T14:06:42.021583+0000","last_clean":"2024-09-18T14:06:42.021583+0000","last_became_active":"2024-09-18T14:06:40.014703+0000","last_became_peered":"2024-09-18T14:06:40.014703+0000","last_unstale":"2024-09-18T14:06:42.021583+0000","last_undegraded":"2024-09-18T14:06:42.021583+0000","last_fullsized":"2024-09-18T14:06:42.021583+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:46:17.562922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048751900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.829521+0000","last_change":"2024-09-18T14:06:42.829873+0000","last_active":"2024-09-18T14:06:42.829521+0000","last_peered":"2024-09-18T14:06:42.829521+0000","last_clean":"2024-09-18T14:06:42.829521+0000","last_became_active":"2024-09-18T14:06:40.014302+0000","last_became_peered":"2024-09-18T14:06:40.014302+0000","last_unstale":"2024-09-18T14:06:42.829521+0000","last_undegraded":"2024-09-18T14:06:42.829521+0000","last_fullsized":"2024-09-18T14:06:42.829521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:51:59.683672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.016348+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:42.016348+0000","last_peered":"2024-09-18T14:06:42.016348+0000","last_clean":"2024-09-18T14:06:42.016348+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:42.016348+0000","last_undegraded":"2024-09-18T14:06:42.016348+0000","last_fullsized":"2024-09-18T14:06:42.016348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.828995+0000","last_change":"2024-09-18T14:06:42.829393+0000","last_active":"2024-09-18T14:06:42.828995+0000","last_peered":"2024-09-18T14:06:42.828995+0000","last_clean":"2024-09-18T14:06:42.828995+0000","last_became_active":"2024-09-18T14:06:40.014431+0000","last_became_peered":"2024-09-18T14:06:40.014431+0000","last_unstale":"2024-09-18T14:06:42.828995+0000","last_undegraded":"2024-09-18T14:06:42.828995+0000","last_fullsized":"2024-09-18T14:06:42.828995+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:27:29.592710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"12'1","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021867+0000","last_change":"2024-09-18T14:06:42.022150+0000","last_active":"2024-09-18T14:06:42.021867+0000","last_peered":"2024-09-18T14:06:42.021867+0000","last_clean":"2024-09-18T14:06:42.021867+0000","last_became_active":"2024-09-18T14:06:40.013037+0000","last_became_peered":"2024-09-18T14:06:40.013037+0000","last_unstale":"2024-09-18T14:06:42.021867+0000","last_undegraded":"2024-09-18T14:06:42.021867+0000","last_fullsized":"2024-09-18T14:06:42.021867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:54:59.496509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070704800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"14'2","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.017768+0000","last_change":"2024-09-18T14:06:42.017768+0000","last_active":"2024-09-18T14:06:42.017768+0000","last_peered":"2024-09-18T14:06:42.017768+0000","last_clean":"2024-09-18T14:06:42.017768+0000","last_became_active":"2024-09-18T14:06:40.015970+0000","last_became_peered":"2024-09-18T14:06:40.015970+0000","last_unstale":"2024-09-18T14:06:42.017768+0000","last_undegraded":"2024-09-18T14:06:42.017768+0000","last_fullsized":"2024-09-18T14:06:42.017768+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:33:09.363728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.829118+0000","last_change":"2024-09-18T14:06:42.829469+0000","last_active":"2024-09-18T14:06:42.829118+0000","last_peered":"2024-09-18T14:06:42.829118+0000","last_clean":"2024-09-18T14:06:42.829118+0000","last_became_active":"2024-09-18T14:06:40.014952+0000","last_became_peered":"2024-09-18T14:06:40.014952+0000","last_unstale":"2024-09-18T14:06:42.829118+0000","last_undegraded":"2024-09-18T14:06:42.829118+0000","last_fullsized":"2024-09-18T14:06:42.829118+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:58:56.331843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021486+0000","last_change":"2024-09-18T14:06:42.021799+0000","last_active":"2024-09-18T14:06:42.021486+0000","last_peered":"2024-09-18T14:06:42.021486+0000","last_clean":"2024-09-18T14:06:42.021486+0000","last_became_active":"2024-09-18T14:06:40.014543+0000","last_became_peered":"2024-09-18T14:06:40.014543+0000","last_unstale":"2024-09-18T14:06:42.021486+0000","last_undegraded":"2024-09-18T14:06:42.021486+0000","last_fullsized":"2024-09-18T14:06:42.021486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:32:23.233193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044037999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705668,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297468,"kb_used_data":112,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074372,"statfs":{"total":96636764160,"available":96332156928,"internally_reserved":0,"allocated":114688,"data_stored":30746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69199999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":34342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69699999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]}]},{"osd":1,"up_from":9,"seq":38654705668,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":126976,"data_stored":34361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":33783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66200000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:45.485 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T14:06:45.485 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-18T14:06:45.485 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-18T14:06:45.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-18T14:06:45.808 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:45.809 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-18T14:06:45.822 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-18T14:06:45.823 INFO:teuthology.run_tasks:Running task osd_recovery.test_incomplete_pgs... 2024-09-18T14:06:45.882 INFO:tasks.osd_recovery:num_osds is 4 2024-09-18T14:06:45.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:06:46.181 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:46.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":14,"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","created":"2024-09-18T14:06:29.125568+0000","modified":"2024-09-18T14:06:42.005576+0000","last_up_change":"2024-09-18T14:06:36.969101+0000","last_in_change":"2024-09-18T14:06:30.498654+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T14:06:38.348847+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"08356eed-f55f-427f-bce6-f522a976a2c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":22294},{"type":"v1","addr":"172.21.15.16:6809","nonce":22294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":22294},{"type":"v1","addr":"172.21.15.16:6811","nonce":22294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":22294},{"type":"v1","addr":"172.21.15.16:6815","nonce":22294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":22294},{"type":"v1","addr":"172.21.15.16:6813","nonce":22294}]},"public_addr":"172.21.15.16:6809/22294","cluster_addr":"172.21.15.16:6811/22294","heartbeat_back_addr":"172.21.15.16:6815/22294","heartbeat_front_addr":"172.21.15.16:6813/22294","state":["exists","up"]},{"osd":1,"uuid":"851cad7b-f44e-48a3-9d7a-2498617c734e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":22293},{"type":"v1","addr":"172.21.15.16:6825","nonce":22293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":22293},{"type":"v1","addr":"172.21.15.16:6827","nonce":22293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":22293},{"type":"v1","addr":"172.21.15.16:6831","nonce":22293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":22293},{"type":"v1","addr":"172.21.15.16:6829","nonce":22293}]},"public_addr":"172.21.15.16:6825/22293","cluster_addr":"172.21.15.16:6827/22293","heartbeat_back_addr":"172.21.15.16:6831/22293","heartbeat_front_addr":"172.21.15.16:6829/22293","state":["exists","up"]},{"osd":2,"uuid":"8dc73a5a-05ad-471a-8d5b-442d5605ab45","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":22295},{"type":"v1","addr":"172.21.15.16:6801","nonce":22295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":22295},{"type":"v1","addr":"172.21.15.16:6803","nonce":22295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":22295},{"type":"v1","addr":"172.21.15.16:6807","nonce":22295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":22295},{"type":"v1","addr":"172.21.15.16:6805","nonce":22295}]},"public_addr":"172.21.15.16:6801/22295","cluster_addr":"172.21.15.16:6803/22295","heartbeat_back_addr":"172.21.15.16:6807/22295","heartbeat_front_addr":"172.21.15.16:6805/22295","state":["exists","up"]},{"osd":3,"uuid":"536c2bb9-0d34-4dc1-b8b2-f2df92543320","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":22310},{"type":"v1","addr":"172.21.15.16:6817","nonce":22310}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":22310},{"type":"v1","addr":"172.21.15.16:6819","nonce":22310}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":22310},{"type":"v1","addr":"172.21.15.16:6823","nonce":22310}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":22310},{"type":"v1","addr":"172.21.15.16:6821","nonce":22310}]},"public_addr":"172.21.15.16:6817/22310","cluster_addr":"172.21.15.16:6819/22310","heartbeat_back_addr":"172.21.15.16:6823/22310","heartbeat_front_addr":"172.21.15.16:6821/22310","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.510803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.952904+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.723520+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.858647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":1,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:06:46.194 INFO:tasks.osd_recovery.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2024-09-18T14:06:38.348847+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '14', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 14, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2024-09-18T14:06:46.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-18T14:06:46.492 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2024-09-18T14:06:46.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-18T14:06:46.786 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 14 2024-09-18T14:06:46.786 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 946c9667-3cd7-48a9-8c9c-fd6b4af766c9 2024-09-18T14:06:46.786 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-18T14:06:29.125568+0000 2024-09-18T14:06:46.787 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-09-18T14:06:42.005576+0000 2024-09-18T14:06:46.787 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-18T14:06:46.787 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 4 2024-09-18T14:06:46.787 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-09-18T14:06:46.787 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-09-18T14:06:46.787 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-09-18T14:06:46.788 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-09-18T14:06:46.788 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client jewel 2024-09-18T14:06:46.788 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release quincy 2024-09-18T14:06:46.788 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-09-18T14:06:46.788 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-18T14:06:46.788 INFO:teuthology.orchestra.run.smithi016.stdout: removed_snaps_queue [2~1] 2024-09-18T14:06:46.788 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 4 2024-09-18T14:06:46.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/22294,v1:172.21.15.16:6809/22294] [v2:172.21.15.16:6810/22294,v1:172.21.15.16:6811/22294] exists,up 08356eed-f55f-427f-bce6-f522a976a2c9 2024-09-18T14:06:46.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6824/22293,v1:172.21.15.16:6825/22293] [v2:172.21.15.16:6826/22293,v1:172.21.15.16:6827/22293] exists,up 851cad7b-f44e-48a3-9d7a-2498617c734e 2024-09-18T14:06:46.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] [v2:172.21.15.16:6802/22295,v1:172.21.15.16:6803/22295] exists,up 8dc73a5a-05ad-471a-8d5b-442d5605ab45 2024-09-18T14:06:46.789 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] [v2:172.21.15.16:6818/22310,v1:172.21.15.16:6819/22310] exists,up 536c2bb9-0d34-4dc1-b8b2-f2df92543320 2024-09-18T14:06:46.800 INFO:tasks.osd_recovery.ceph_manager:['osd.0 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/22294,v1:172.21.15.16:6809/22294] [v2:172.21.15.16:6810/22294,v1:172.21.15.16:6811/22294] exists,up 08356eed-f55f-427f-bce6-f522a976a2c9', 'osd.1 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6824/22293,v1:172.21.15.16:6825/22293] [v2:172.21.15.16:6826/22293,v1:172.21.15.16:6827/22293] exists,up 851cad7b-f44e-48a3-9d7a-2498617c734e', 'osd.2 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] [v2:172.21.15.16:6802/22295,v1:172.21.15.16:6803/22295] exists,up 8dc73a5a-05ad-471a-8d5b-442d5605ab45', 'osd.3 up in weight 1 up_from 9 up_thru 11 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] [v2:172.21.15.16:6818/22310,v1:172.21.15.16:6819/22310] exists,up 536c2bb9-0d34-4dc1-b8b2-f2df92543320'] 2024-09-18T14:06:46.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T14:06:46.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T14:06:46.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T14:06:46.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T14:06:46.968 INFO:teuthology.orchestra.run.smithi016.stdout:38654705669 2024-09-18T14:06:46.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:06:46.974 INFO:teuthology.orchestra.run.smithi016.stdout:38654705669 2024-09-18T14:06:46.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:06:46.978 INFO:teuthology.orchestra.run.smithi016.stdout:38654705669 2024-09-18T14:06:46.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:06:46.982 INFO:teuthology.orchestra.run.smithi016.stdout:38654705669 2024-09-18T14:06:46.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:06:47.249 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:47.265 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:47.266 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705668 for osd.3 2024-09-18T14:06:47.279 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705668 for osd.1 2024-09-18T14:06:47.286 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:47.304 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705668 for osd.2 2024-09-18T14:06:47.315 INFO:teuthology.orchestra.run.smithi016.stdout:38654705668 2024-09-18T14:06:47.328 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705668 for osd.0 2024-09-18T14:06:48.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:06:48.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:06:48.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:06:48.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:06:48.624 INFO:teuthology.orchestra.run.smithi016.stdout:38654705669 2024-09-18T14:06:48.627 INFO:teuthology.orchestra.run.smithi016.stdout:38654705669 2024-09-18T14:06:48.639 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705669 for osd.0 2024-09-18T14:06:48.640 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:48.642 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705669 for osd.3 2024-09-18T14:06:48.672 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:48.673 INFO:teuthology.orchestra.run.smithi016.stdout:38654705670 2024-09-18T14:06:48.674 INFO:teuthology.orchestra.run.smithi016.stdout:38654705669 2024-09-18T14:06:48.674 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705669 for osd.2 2024-09-18T14:06:48.674 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:48.675 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705669 got 38654705670 for osd.1 2024-09-18T14:06:48.675 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:48.675 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2024-09-18T14:06:48.675 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:48.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:48.964 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:48.974 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-18T14:06:48.856879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189924,"kb_used_data":484,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297436,"statfs":{"total":386547056640,"available":385328574464,"internally_reserved":0,"allocated":495616,"data_stored":136828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.842642"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021583+0000","last_change":"2024-09-18T14:06:42.021844+0000","last_active":"2024-09-18T14:06:42.021583+0000","last_peered":"2024-09-18T14:06:42.021583+0000","last_clean":"2024-09-18T14:06:42.021583+0000","last_became_active":"2024-09-18T14:06:40.014703+0000","last_became_peered":"2024-09-18T14:06:40.014703+0000","last_unstale":"2024-09-18T14:06:42.021583+0000","last_undegraded":"2024-09-18T14:06:42.021583+0000","last_fullsized":"2024-09-18T14:06:42.021583+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:46:17.562922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048751900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.829521+0000","last_change":"2024-09-18T14:06:42.829873+0000","last_active":"2024-09-18T14:06:42.829521+0000","last_peered":"2024-09-18T14:06:42.829521+0000","last_clean":"2024-09-18T14:06:42.829521+0000","last_became_active":"2024-09-18T14:06:40.014302+0000","last_became_peered":"2024-09-18T14:06:40.014302+0000","last_unstale":"2024-09-18T14:06:42.829521+0000","last_undegraded":"2024-09-18T14:06:42.829521+0000","last_fullsized":"2024-09-18T14:06:42.829521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:51:59.683672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.016348+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:42.016348+0000","last_peered":"2024-09-18T14:06:42.016348+0000","last_clean":"2024-09-18T14:06:42.016348+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:42.016348+0000","last_undegraded":"2024-09-18T14:06:42.016348+0000","last_fullsized":"2024-09-18T14:06:42.016348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.828995+0000","last_change":"2024-09-18T14:06:42.829393+0000","last_active":"2024-09-18T14:06:42.828995+0000","last_peered":"2024-09-18T14:06:42.828995+0000","last_clean":"2024-09-18T14:06:42.828995+0000","last_became_active":"2024-09-18T14:06:40.014431+0000","last_became_peered":"2024-09-18T14:06:40.014431+0000","last_unstale":"2024-09-18T14:06:42.828995+0000","last_undegraded":"2024-09-18T14:06:42.828995+0000","last_fullsized":"2024-09-18T14:06:42.828995+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:27:29.592710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"12'1","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021867+0000","last_change":"2024-09-18T14:06:42.022150+0000","last_active":"2024-09-18T14:06:42.021867+0000","last_peered":"2024-09-18T14:06:42.021867+0000","last_clean":"2024-09-18T14:06:42.021867+0000","last_became_active":"2024-09-18T14:06:40.013037+0000","last_became_peered":"2024-09-18T14:06:40.013037+0000","last_unstale":"2024-09-18T14:06:42.021867+0000","last_undegraded":"2024-09-18T14:06:42.021867+0000","last_fullsized":"2024-09-18T14:06:42.021867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:54:59.496509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070704800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"14'2","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.017768+0000","last_change":"2024-09-18T14:06:42.017768+0000","last_active":"2024-09-18T14:06:42.017768+0000","last_peered":"2024-09-18T14:06:42.017768+0000","last_clean":"2024-09-18T14:06:42.017768+0000","last_became_active":"2024-09-18T14:06:40.015970+0000","last_became_peered":"2024-09-18T14:06:40.015970+0000","last_unstale":"2024-09-18T14:06:42.017768+0000","last_undegraded":"2024-09-18T14:06:42.017768+0000","last_fullsized":"2024-09-18T14:06:42.017768+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:33:09.363728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.829118+0000","last_change":"2024-09-18T14:06:42.829469+0000","last_active":"2024-09-18T14:06:42.829118+0000","last_peered":"2024-09-18T14:06:42.829118+0000","last_clean":"2024-09-18T14:06:42.829118+0000","last_became_active":"2024-09-18T14:06:40.014952+0000","last_became_peered":"2024-09-18T14:06:40.014952+0000","last_unstale":"2024-09-18T14:06:42.829118+0000","last_undegraded":"2024-09-18T14:06:42.829118+0000","last_fullsized":"2024-09-18T14:06:42.829118+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:58:56.331843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021486+0000","last_change":"2024-09-18T14:06:42.021799+0000","last_active":"2024-09-18T14:06:42.021486+0000","last_peered":"2024-09-18T14:06:42.021486+0000","last_clean":"2024-09-18T14:06:42.021486+0000","last_became_active":"2024-09-18T14:06:40.014543+0000","last_became_peered":"2024-09-18T14:06:40.014543+0000","last_unstale":"2024-09-18T14:06:42.021486+0000","last_undegraded":"2024-09-18T14:06:42.021486+0000","last_fullsized":"2024-09-18T14:06:42.021486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:32:23.233193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044037999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":34342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":34342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":126976,"data_stored":34361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":33783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:48.975 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:49.260 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:49.263 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:49.273 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-18T14:06:48.856879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":377487360,"kb_used":1189924,"kb_used_data":484,"kb_used_omap":6,"kb_used_meta":1189369,"kb_avail":376297436,"statfs":{"total":386547056640,"available":385328574464,"internally_reserved":0,"allocated":495616,"data_stored":136828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6816,"internal_metadata":1217914208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.842642"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021583+0000","last_change":"2024-09-18T14:06:42.021844+0000","last_active":"2024-09-18T14:06:42.021583+0000","last_peered":"2024-09-18T14:06:42.021583+0000","last_clean":"2024-09-18T14:06:42.021583+0000","last_became_active":"2024-09-18T14:06:40.014703+0000","last_became_peered":"2024-09-18T14:06:40.014703+0000","last_unstale":"2024-09-18T14:06:42.021583+0000","last_undegraded":"2024-09-18T14:06:42.021583+0000","last_fullsized":"2024-09-18T14:06:42.021583+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:46:17.562922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048751900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.6","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.829521+0000","last_change":"2024-09-18T14:06:42.829873+0000","last_active":"2024-09-18T14:06:42.829521+0000","last_peered":"2024-09-18T14:06:42.829521+0000","last_clean":"2024-09-18T14:06:42.829521+0000","last_became_active":"2024-09-18T14:06:40.014302+0000","last_became_peered":"2024-09-18T14:06:40.014302+0000","last_unstale":"2024-09-18T14:06:42.829521+0000","last_undegraded":"2024-09-18T14:06:42.829521+0000","last_fullsized":"2024-09-18T14:06:42.829521+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:51:59.683672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.5","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.016348+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:42.016348+0000","last_peered":"2024-09-18T14:06:42.016348+0000","last_clean":"2024-09-18T14:06:42.016348+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:42.016348+0000","last_undegraded":"2024-09-18T14:06:42.016348+0000","last_fullsized":"2024-09-18T14:06:42.016348+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.4","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.828995+0000","last_change":"2024-09-18T14:06:42.829393+0000","last_active":"2024-09-18T14:06:42.828995+0000","last_peered":"2024-09-18T14:06:42.828995+0000","last_clean":"2024-09-18T14:06:42.828995+0000","last_became_active":"2024-09-18T14:06:40.014431+0000","last_became_peered":"2024-09-18T14:06:40.014431+0000","last_unstale":"2024-09-18T14:06:42.828995+0000","last_undegraded":"2024-09-18T14:06:42.828995+0000","last_fullsized":"2024-09-18T14:06:42.828995+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:27:29.592710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.3","version":"12'1","reported_seq":15,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021867+0000","last_change":"2024-09-18T14:06:42.022150+0000","last_active":"2024-09-18T14:06:42.021867+0000","last_peered":"2024-09-18T14:06:42.021867+0000","last_clean":"2024-09-18T14:06:42.021867+0000","last_became_active":"2024-09-18T14:06:40.013037+0000","last_became_peered":"2024-09-18T14:06:40.013037+0000","last_unstale":"2024-09-18T14:06:42.021867+0000","last_undegraded":"2024-09-18T14:06:42.021867+0000","last_fullsized":"2024-09-18T14:06:42.021867+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:54:59.496509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070704800000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.2","version":"14'2","reported_seq":16,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.017768+0000","last_change":"2024-09-18T14:06:42.017768+0000","last_active":"2024-09-18T14:06:42.017768+0000","last_peered":"2024-09-18T14:06:42.017768+0000","last_clean":"2024-09-18T14:06:42.017768+0000","last_became_active":"2024-09-18T14:06:40.015970+0000","last_became_peered":"2024-09-18T14:06:40.015970+0000","last_unstale":"2024-09-18T14:06:42.017768+0000","last_undegraded":"2024-09-18T14:06:42.017768+0000","last_fullsized":"2024-09-18T14:06:42.017768+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:33:09.363728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.1","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.829118+0000","last_change":"2024-09-18T14:06:42.829469+0000","last_active":"2024-09-18T14:06:42.829118+0000","last_peered":"2024-09-18T14:06:42.829118+0000","last_clean":"2024-09-18T14:06:42.829118+0000","last_became_active":"2024-09-18T14:06:40.014952+0000","last_became_peered":"2024-09-18T14:06:40.014952+0000","last_unstale":"2024-09-18T14:06:42.829118+0000","last_undegraded":"2024-09-18T14:06:42.829118+0000","last_fullsized":"2024-09-18T14:06:42.829118+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:58:56.331843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"0'0","reported_seq":14,"reported_epoch":14,"state":"active+clean","last_fresh":"2024-09-18T14:06:42.021486+0000","last_change":"2024-09-18T14:06:42.021799+0000","last_active":"2024-09-18T14:06:42.021486+0000","last_peered":"2024-09-18T14:06:42.021486+0000","last_clean":"2024-09-18T14:06:42.021486+0000","last_became_active":"2024-09-18T14:06:40.014543+0000","last_became_peered":"2024-09-18T14:06:40.014543+0000","last_unstale":"2024-09-18T14:06:42.021486+0000","last_undegraded":"2024-09-18T14:06:42.021486+0000","last_fullsized":"2024-09-18T14:06:42.021486+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:32:23.233193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044037999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":34342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.623}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80800000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705670,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":34342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61499999999999999}]}]},{"osd":1,"up_from":9,"seq":38654705670,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":124,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":126976,"data_stored":34361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705670,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297480,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074360,"statfs":{"total":96636764160,"available":96332144640,"internally_reserved":0,"allocated":122880,"data_stored":33783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:49.274 INFO:tasks.osd_recovery.ceph_manager:clean! 2024-09-18T14:06:49.274 INFO:tasks.osd_recovery:Testing incomplete pgs... 2024-09-18T14:06:49.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set osd_recovery_delay_start 1000 2024-09-18T14:06:49.379 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:06:49.379 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_recovery_delay_start = '' " 2024-09-18T14:06:49.379 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:06:49.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok config set osd_recovery_delay_start 1000 2024-09-18T14:06:49.491 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:06:49.491 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_recovery_delay_start = '' " 2024-09-18T14:06:49.491 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:06:49.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set osd_recovery_delay_start 1000 2024-09-18T14:06:49.600 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:06:49.600 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_recovery_delay_start = '' " 2024-09-18T14:06:49.600 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:06:49.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set osd_recovery_delay_start 1000 2024-09-18T14:06:49.713 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:06:49.713 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_recovery_delay_start = '' " 2024-09-18T14:06:49.713 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:06:49.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 1 2024-09-18T14:06:50.025 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-18T14:06:50.019+0000 7f755a98d700 -1 mon.a@0(leader).osd e15 definitely_dead 0 2024-09-18T14:06:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:marked out osd.0. marked out osd.1. 2024-09-18T14:06:50.061 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T14:06:50.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T14:06:50.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T14:06:50.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T14:06:50.217 INFO:teuthology.orchestra.run.smithi016.stdout:38654705671 2024-09-18T14:06:50.217 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:50.218 INFO:teuthology.orchestra.run.smithi016.stdout:38654705671 2024-09-18T14:06:50.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:06:50.231 INFO:teuthology.orchestra.run.smithi016.stdout:38654705671 2024-09-18T14:06:50.232 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:50.233 INFO:teuthology.orchestra.run.smithi016.stdout:38654705671 2024-09-18T14:06:50.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:06:50.510 INFO:teuthology.orchestra.run.smithi016.stdout:38654705670 2024-09-18T14:06:50.518 INFO:teuthology.orchestra.run.smithi016.stdout:38654705670 2024-09-18T14:06:50.524 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705671 got 38654705670 for osd.2 2024-09-18T14:06:50.537 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705671 got 38654705670 for osd.3 2024-09-18T14:06:51.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:06:51.537 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:06:51.815 INFO:teuthology.orchestra.run.smithi016.stdout:38654705670 2024-09-18T14:06:51.834 INFO:teuthology.orchestra.run.smithi016.stdout:38654705670 2024-09-18T14:06:51.835 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705671 got 38654705670 for osd.2 2024-09-18T14:06:51.846 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705671 got 38654705670 for osd.3 2024-09-18T14:06:52.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-18T14:06:52.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-18T14:06:53.150 INFO:teuthology.orchestra.run.smithi016.stdout:38654705671 2024-09-18T14:06:53.164 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705671 got 38654705671 for osd.2 2024-09-18T14:06:53.164 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:53.171 INFO:teuthology.orchestra.run.smithi016.stdout:38654705671 2024-09-18T14:06:53.186 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705671 got 38654705671 for osd.3 2024-09-18T14:06:53.186 DEBUG:teuthology.parallel:result is None 2024-09-18T14:06:53.186 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2024-09-18T14:06:53.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:53.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:53.475 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:53.485 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-18T14:06:51.905602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594968,"kb_used_data":256,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148712,"statfs":{"total":193273528320,"available":192664281088,"internally_reserved":0,"allocated":262144,"data_stored":75144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001448"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":4,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.043509+0000","last_change":"2024-09-18T14:06:50.043725+0000","last_active":"2024-09-18T14:06:38.985939+0000","last_peered":"2024-09-18T14:06:38.985939+0000","last_clean":"2024-09-18T14:06:38.985939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T14:06:50.043509+0000","last_undegraded":"2024-09-18T14:06:50.043509+0000","last_fullsized":"2024-09-18T14:06:50.043509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:23:08.616580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.045396+0000","last_change":"2024-09-18T14:06:50.045634+0000","last_active":"2024-09-18T14:06:50.045367+0000","last_peered":"2024-09-18T14:06:49.639808+0000","last_clean":"2024-09-18T14:06:49.639808+0000","last_became_active":"2024-09-18T14:06:40.014302+0000","last_became_peered":"2024-09-18T14:06:40.014302+0000","last_unstale":"2024-09-18T14:06:50.045396+0000","last_undegraded":"2024-09-18T14:06:50.045396+0000","last_fullsized":"2024-09-18T14:06:50.045396+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:57:39.306161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T14:06:48.861117+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:48.861117+0000","last_peered":"2024-09-18T14:06:48.861117+0000","last_clean":"2024-09-18T14:06:48.861117+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:48.861117+0000","last_undegraded":"2024-09-18T14:06:48.861117+0000","last_fullsized":"2024-09-18T14:06:48.861117+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.045266+0000","last_change":"2024-09-18T14:06:50.045484+0000","last_active":"2024-09-18T14:06:50.045235+0000","last_peered":"2024-09-18T14:06:49.639852+0000","last_clean":"2024-09-18T14:06:49.639852+0000","last_became_active":"2024-09-18T14:06:40.014431+0000","last_became_peered":"2024-09-18T14:06:40.014431+0000","last_unstale":"2024-09-18T14:06:50.045266+0000","last_undegraded":"2024-09-18T14:06:50.045266+0000","last_fullsized":"2024-09-18T14:06:50.045266+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:55:21.840948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":10,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.040334+0000","last_change":"2024-09-18T14:06:50.040643+0000","last_active":"2024-09-18T14:06:50.040291+0000","last_peered":"2024-09-18T14:06:40.013417+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:40.013037+0000","last_became_peered":"2024-09-18T14:06:40.013037+0000","last_unstale":"2024-09-18T14:06:50.040334+0000","last_undegraded":"2024-09-18T14:06:50.040334+0000","last_fullsized":"2024-09-18T14:06:50.040334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:51:24.349437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.043453+0000","last_change":"2024-09-18T14:06:50.043453+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T14:06:50.043453+0000","last_undegraded":"2024-09-18T14:06:50.043453+0000","last_fullsized":"2024-09-18T14:06:50.043453+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:56:09.698119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.045191+0000","last_change":"2024-09-18T14:06:50.045420+0000","last_active":"2024-09-18T14:06:50.045160+0000","last_peered":"2024-09-18T14:06:49.639676+0000","last_clean":"2024-09-18T14:06:49.639676+0000","last_became_active":"2024-09-18T14:06:40.014952+0000","last_became_peered":"2024-09-18T14:06:40.014952+0000","last_unstale":"2024-09-18T14:06:50.045191+0000","last_undegraded":"2024-09-18T14:06:50.045191+0000","last_fullsized":"2024-09-18T14:06:50.045191+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:47:00.910857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.041135+0000","last_change":"2024-09-18T14:06:50.041135+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T14:06:50.041135+0000","last_undegraded":"2024-09-18T14:06:50.041135+0000","last_fullsized":"2024-09-18T14:06:50.041135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:30:23.015219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":37572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":37572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:53.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:53.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:53.784 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:53.794 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-18T14:06:51.905602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594968,"kb_used_data":256,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148712,"statfs":{"total":193273528320,"available":192664281088,"internally_reserved":0,"allocated":262144,"data_stored":75144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-1,"num_object_clones":0,"num_object_copies":-2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-2,"num_write_kb":-2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001448"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":4,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.043509+0000","last_change":"2024-09-18T14:06:50.043725+0000","last_active":"2024-09-18T14:06:38.985939+0000","last_peered":"2024-09-18T14:06:38.985939+0000","last_clean":"2024-09-18T14:06:38.985939+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T14:06:50.043509+0000","last_undegraded":"2024-09-18T14:06:50.043509+0000","last_fullsized":"2024-09-18T14:06:50.043509+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T14:23:08.616580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.045396+0000","last_change":"2024-09-18T14:06:50.045634+0000","last_active":"2024-09-18T14:06:50.045367+0000","last_peered":"2024-09-18T14:06:49.639808+0000","last_clean":"2024-09-18T14:06:49.639808+0000","last_became_active":"2024-09-18T14:06:40.014302+0000","last_became_peered":"2024-09-18T14:06:40.014302+0000","last_unstale":"2024-09-18T14:06:50.045396+0000","last_undegraded":"2024-09-18T14:06:50.045396+0000","last_fullsized":"2024-09-18T14:06:50.045396+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:57:39.306161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-18T14:06:48.861117+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:48.861117+0000","last_peered":"2024-09-18T14:06:48.861117+0000","last_clean":"2024-09-18T14:06:48.861117+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:48.861117+0000","last_undegraded":"2024-09-18T14:06:48.861117+0000","last_fullsized":"2024-09-18T14:06:48.861117+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.045266+0000","last_change":"2024-09-18T14:06:50.045484+0000","last_active":"2024-09-18T14:06:50.045235+0000","last_peered":"2024-09-18T14:06:49.639852+0000","last_clean":"2024-09-18T14:06:49.639852+0000","last_became_active":"2024-09-18T14:06:40.014431+0000","last_became_peered":"2024-09-18T14:06:40.014431+0000","last_unstale":"2024-09-18T14:06:50.045266+0000","last_undegraded":"2024-09-18T14:06:50.045266+0000","last_fullsized":"2024-09-18T14:06:50.045266+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:55:21.840948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":10,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.040334+0000","last_change":"2024-09-18T14:06:50.040643+0000","last_active":"2024-09-18T14:06:50.040291+0000","last_peered":"2024-09-18T14:06:40.013417+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:40.013037+0000","last_became_peered":"2024-09-18T14:06:40.013037+0000","last_unstale":"2024-09-18T14:06:50.040334+0000","last_undegraded":"2024-09-18T14:06:50.040334+0000","last_fullsized":"2024-09-18T14:06:50.040334+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:51:24.349437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.043453+0000","last_change":"2024-09-18T14:06:50.043453+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T14:06:50.043453+0000","last_undegraded":"2024-09-18T14:06:50.043453+0000","last_fullsized":"2024-09-18T14:06:50.043453+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:56:09.698119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":19,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.045191+0000","last_change":"2024-09-18T14:06:50.045420+0000","last_active":"2024-09-18T14:06:50.045160+0000","last_peered":"2024-09-18T14:06:49.639676+0000","last_clean":"2024-09-18T14:06:49.639676+0000","last_became_active":"2024-09-18T14:06:40.014952+0000","last_became_peered":"2024-09-18T14:06:40.014952+0000","last_unstale":"2024-09-18T14:06:50.045191+0000","last_undegraded":"2024-09-18T14:06:50.045191+0000","last_fullsized":"2024-09-18T14:06:50.045191+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:47:00.910857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,2],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":1,"reported_epoch":16,"state":"peering","last_fresh":"2024-09-18T14:06:50.041135+0000","last_change":"2024-09-18T14:06:50.041135+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-18T14:06:50.041135+0000","last_undegraded":"2024-09-18T14:06:50.041135+0000","last_fullsized":"2024-09-18T14:06:50.041135+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:30:23.015219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705671,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":37572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705671,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297484,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074356,"statfs":{"total":96636764160,"available":96332140544,"internally_reserved":0,"allocated":131072,"data_stored":37572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:53.796 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:54.090 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:54.094 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:54.109 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-18T14:06:53.906075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001408"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:54.110 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:06:54.110 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:06:54.110 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:06:54.110 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:06:54.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:06:54.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:54.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":25,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:06:54.476 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:06:54.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:54.757 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:54.760 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:54.775 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-18T14:06:53.906075+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001408"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:57.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:58.075 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:58.090 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-18T14:06:57.907031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001858"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:58.090 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:58.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:58.374 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-18T14:06:57.907031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001858"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:06:58.390 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:06:58.390 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:06:58.391 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:06:58.391 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:06:58.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:06:58.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:58.733 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":29,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:06:58.746 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:06:58.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:06:59.032 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:06:59.033 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:06:59.044 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-18T14:06:57.907031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":7,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001858"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705672,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705672,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:02.045 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:02.342 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:02.345 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:02.356 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-18T14:07:01.908043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.869725"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:02.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:02.637 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:02.640 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:02.651 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-18T14:07:01.908043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.869725"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:02.651 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:02.651 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:02.652 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:02.652 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:02.652 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:02.997 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:02.997 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":33,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75,"write_bytes_sec":188,"read_op_per_sec":0,"write_op_per_sec":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:03.011 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:03.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:03.287 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:03.290 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:03.300 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-09-18T14:07:01.908043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.869725"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705673,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705673,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.746}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86599999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:06.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:06.600 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:06.603 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:06.614 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-18T14:07:05.909241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:06.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:06.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:06.901 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:06.918 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-18T14:07:05.909241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:06.918 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:06.919 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:06.919 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:06.919 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:06.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:07.262 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":38,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:07.276 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:07.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:07.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:07.560 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:07.570 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2024-09-18T14:07:05.909241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705674,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64900000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705674,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:10.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:10.874 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:10.877 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:10.892 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-18T14:07:09.910415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:10.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:11.181 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:11.184 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:11.201 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-18T14:07:09.910415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:11.201 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:11.202 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:11.202 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:11.202 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:11.202 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:11.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:11.549 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":42,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:11.564 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:11.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:11.849 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:11.852 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:11.863 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-18T14:07:09.910415+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66500000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70799999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705675,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70099999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:14.864 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:15.164 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:15.167 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:15.178 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-09-18T14:07:13.911732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:15.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:15.472 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:15.475 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:15.485 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2024-09-18T14:07:13.911732+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:15.485 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:15.485 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:15.485 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:15.486 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:15.486 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:15.823 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:15.823 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":46,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:15.836 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:15.836 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:16.125 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:16.128 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:16.139 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-18T14:07:15.912297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003682"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:19.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:19.442 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:19.442 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:19.452 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-09-18T14:07:17.912825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003583"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:19.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:19.736 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:19.739 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:19.750 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-09-18T14:07:17.912825+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003583"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705676,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:19.751 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:19.751 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:19.751 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:19.751 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:19.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:20.098 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:20.098 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":50,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:20.113 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:20.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:20.392 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:20.395 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:20.405 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2024-09-18T14:07:19.913347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003511"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:23.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:23.709 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:23.712 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:23.723 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-09-18T14:07:21.913856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003440"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705677,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85399999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:23.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:24.012 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:24.025 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-18T14:07:23.914407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:24.026 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:24.026 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:24.026 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:24.026 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:24.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:24.379 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:24.379 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":55,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:24.393 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:24.394 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:24.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:24.682 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:24.692 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":37,"stamp":"2024-09-18T14:07:23.914407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:27.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:27.998 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:28.001 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:28.012 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-18T14:07:27.915499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:28.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:28.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:28.301 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:28.312 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-18T14:07:27.915499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:28.312 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:28.312 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:28.313 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:28.313 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:28.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:28.653 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:28.653 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":59,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:28.667 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:28.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:28.953 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:28.955 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:28.966 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-09-18T14:07:27.915499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003202"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705678,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65900000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88200000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:31.968 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:32.265 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:32.268 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:32.278 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-18T14:07:31.916582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003235"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:32.279 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:32.559 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:32.562 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:32.573 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-18T14:07:31.916582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003235"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:32.574 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:32.574 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:32.574 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:32.574 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:32.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:32.920 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:32.920 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":63,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":1,"modified":"2024-09-18T14:06:29.127480+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:32.938 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:32.938 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:33.224 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:33.227 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:33.238 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":41,"stamp":"2024-09-18T14:07:31.916582+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003235"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80700000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705679,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68999999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92000000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81799999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705679,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:36.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:36.533 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:36.536 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:36.546 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-18T14:07:35.917709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]}]},{"osd":1,"up_from":0,"seq":38654705680,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705680,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:36.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:36.836 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:36.851 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-18T14:07:35.917709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]}]},{"osd":1,"up_from":0,"seq":38654705680,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705680,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:36.851 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:36.852 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:36.852 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:36.852 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:36.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:37.194 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:37.194 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":68,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":2,"modified":"2024-09-18T14:07:35.917777+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T14:07:37.213 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:37.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:37.494 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:37.497 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:37.507 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-18T14:07:35.917709+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705680,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.875}]}]},{"osd":1,"up_from":0,"seq":38654705680,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705680,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:40.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:40.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:40.824 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-18T14:07:39.918830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:40.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:41.110 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:41.113 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:41.124 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-18T14:07:39.918830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:41.124 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:41.125 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:41.125 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:41.125 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:41.125 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:41.469 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:41.470 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":72,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:41.484 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:41.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:41.771 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:41.774 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:41.784 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2024-09-18T14:07:39.918830+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705681,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73499999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705681,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:44.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:45.086 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:45.089 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:45.099 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-09-18T14:07:43.920006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003423"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70199999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.77300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.89500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:45.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:45.385 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:45.403 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2024-09-18T14:07:43.920006+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003423"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70199999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.77300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.89500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:45.403 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:45.404 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:45.404 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:45.404 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:45.404 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:45.757 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:45.757 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":76,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:45.777 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:45.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:46.065 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:46.078 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":48,"stamp":"2024-09-18T14:07:45.920558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003428"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70199999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.77300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.89500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:49.079 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:49.376 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:49.379 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:49.390 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2024-09-18T14:07:47.921008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003298"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70199999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.77300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.89500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:49.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:49.672 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:49.675 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:49.686 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":49,"stamp":"2024-09-18T14:07:47.921008+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003298"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70199999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.77300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705682,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.89500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:49.687 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:49.687 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:49.687 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:49.687 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:49.688 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:50.036 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:50.037 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":80,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:50.055 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:50.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:50.338 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:50.412 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:50.412 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":50,"stamp":"2024-09-18T14:07:49.921540+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.79000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61599999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60799999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.75600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:53.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:53.710 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:53.723 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2024-09-18T14:07:51.922090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003260"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.79000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61599999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705683,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60799999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.75600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705683,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:53.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:54.015 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:54.019 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:54.029 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-18T14:07:53.922615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003173"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.39300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.623},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72899999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:54.030 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:54.030 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:54.030 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:54.030 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:54.031 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:54.373 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:54.373 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":85,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:54.386 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:54.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:54.671 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:54.674 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:54.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":52,"stamp":"2024-09-18T14:07:53.922615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003173"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.39300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.623},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72899999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:57.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:57.996 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:58.007 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-18T14:07:57.923739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003180"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.39300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.623},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72899999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:58.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:58.297 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:58.308 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-18T14:07:57.923739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003180"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.39300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.623},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72899999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:07:58.308 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:07:58.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:07:58.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:07:58.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:07:58.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:07:58.659 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:58.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":89,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:07:58.673 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:07:58.673 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:07:58.965 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:07:58.968 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:07:58.979 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2024-09-18T14:07:57.923739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003180"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705684,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.39300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.623},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72899999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705684,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:01.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:02.288 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:02.301 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-09-18T14:08:01.924909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003369"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.41199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68799999999999994}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.78900000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:02.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:02.588 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:02.591 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:02.601 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-09-18T14:08:01.924909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003369"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.41199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68799999999999994}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.78900000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:02.602 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:02.602 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:02.602 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:02.602 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:02.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:02.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:02.956 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":93,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:02.969 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:02.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:03.255 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:03.258 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:03.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":56,"stamp":"2024-09-18T14:08:01.924909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003369"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.41199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705685,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.68799999999999994}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.78900000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705685,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:06.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:06.573 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:06.576 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:06.587 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2024-09-18T14:08:05.926128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003513"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72899999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:06.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:06.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:06.879 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:06.890 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2024-09-18T14:08:05.926128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003513"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72899999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:06.891 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:06.891 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:06.891 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:06.892 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:06.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:07.237 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:07.238 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":98,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:07.252 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:07.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:07.543 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:07.545 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:07.557 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2024-09-18T14:08:05.926128+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003513"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705686,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72899999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:10.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:10.863 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:10.866 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:10.876 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-09-18T14:08:09.927219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.755}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705687,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705687,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:10.877 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:11.168 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:11.171 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:11.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-09-18T14:08:09.927219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.755}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705687,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705687,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:11.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:11.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:11.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:11.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:11.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:11.540 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:11.540 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":102,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:11.553 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:11.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:11.845 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:11.847 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:11.857 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2024-09-18T14:08:09.927219+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.755}]}]},{"osd":2,"up_from":9,"seq":38654705687,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.57599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705687,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705687,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:14.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:15.157 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:15.160 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:15.175 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-18T14:08:13.928365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003456"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.74199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.80700000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:15.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:15.459 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:15.462 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:15.472 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2024-09-18T14:08:13.928365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003456"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.74199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.80700000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:15.473 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:15.473 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:15.474 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:15.474 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:15.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:15.819 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:15.819 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":106,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:15.833 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:15.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:16.117 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:16.119 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:16.130 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-09-18T14:08:15.928911+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.74199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.80700000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:19.131 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:19.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:19.432 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:19.442 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-09-18T14:08:17.929436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.74199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.80700000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:19.443 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:19.726 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:19.740 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":64,"stamp":"2024-09-18T14:08:17.929436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.65700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.74199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705688,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.80700000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705688,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:19.741 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:19.741 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:19.741 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:19.741 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:19.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:20.095 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:20.095 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":110,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:20.113 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:20.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:20.398 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:20.402 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:20.412 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":65,"stamp":"2024-09-18T14:08:19.929948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003292"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79800000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.66500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56899999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.69999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:23.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:23.717 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:23.720 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:23.730 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2024-09-18T14:08:21.930444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003225"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.59299999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79800000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705689,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.66500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.56899999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.69999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705689,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:23.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:24.021 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:24.034 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-09-18T14:08:23.930964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003168"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:24.035 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:24.035 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:24.035 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:24.035 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:24.035 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:24.383 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:24.383 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":115,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:24.396 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:24.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:24.682 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:24.684 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:24.701 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":67,"stamp":"2024-09-18T14:08:23.930964+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003168"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:27.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:28.000 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:28.003 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:28.015 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-18T14:08:27.932005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:28.016 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:28.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:28.306 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:28.317 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-18T14:08:27.932005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:28.318 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:28.318 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:28.318 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:28.318 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:28.318 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:28.663 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:28.664 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":119,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:28.678 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:28.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:28.962 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:28.966 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:28.976 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":69,"stamp":"2024-09-18T14:08:27.932005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705690,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:31.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:32.275 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:32.279 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:32.288 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-09-18T14:08:31.933032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.752},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.84799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70399999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.80400000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:32.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:32.573 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:32.575 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:32.586 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-09-18T14:08:31.933032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.752},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.84799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70399999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.80400000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:32.587 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:32.587 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:32.588 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:32.588 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:32.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:32.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:32.936 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":123,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:32.950 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:32.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:33.245 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:33.248 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:33.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":71,"stamp":"2024-09-18T14:08:31.933032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.504},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705691,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.752},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.84799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70399999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.80400000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705691,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:36.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:36.558 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:36.561 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:36.571 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-09-18T14:08:35.934151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.55800000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.58499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:36.572 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:36.858 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:36.872 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-09-18T14:08:35.934151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.55800000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.58499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:36.873 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:36.873 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:36.873 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:36.873 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:36.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:37.221 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:37.221 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":128,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:37.235 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:37.235 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:37.520 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:37.523 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:37.534 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2024-09-18T14:08:35.934151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.55800000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.70499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705692,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.58499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.72499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705692,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:40.535 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:40.832 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:40.835 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:40.849 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-18T14:08:39.935286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003281"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.70299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66700000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.89300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705693,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705693,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:40.850 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:41.135 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:41.138 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:41.148 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-18T14:08:39.935286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003281"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.70299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66700000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.89300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705693,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705693,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:41.149 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:41.149 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:41.149 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:41.149 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:41.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:41.501 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:41.501 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":132,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:41.515 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:41.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:41.803 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:41.806 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:41.816 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":75,"stamp":"2024-09-18T14:08:39.935286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003281"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.62,"5min":0.62,"15min":0.62},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.70299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66700000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:07:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.73899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705693,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.67500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.89300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705693,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705693,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:44.818 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:45.128 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-18T14:08:43.936448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73899999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:45.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:45.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:45.411 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:45.422 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-18T14:08:43.936448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73899999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:45.423 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:45.423 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:45.423 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:45.424 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:45.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:45.765 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:45.765 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":136,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:45.778 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:45.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:46.065 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:46.078 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-18T14:08:45.937014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003437"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73899999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:49.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:49.383 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:49.386 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:49.397 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-18T14:08:47.937499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003348"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73899999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705695,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:49.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:49.676 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:49.694 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-18T14:08:47.937499+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003348"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.77800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63900000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705694,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.52000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:07:39 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.222,"5min":1.222,"15min":1.222},"last":0.73899999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705695,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705694,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:49.695 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:49.695 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:49.695 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:49.695 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:49.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:50.043 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":140,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:50.057 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:50.058 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:50.344 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:50.347 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:50.357 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":80,"stamp":"2024-09-18T14:08:49.938013+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.81999999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.76400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.746}]}]},{"osd":1,"up_from":0,"seq":38654705695,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705695,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:53.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:53.662 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:53.675 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":81,"stamp":"2024-09-18T14:08:51.938586+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003300"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.81999999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.76400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705695,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.746}]}]},{"osd":1,"up_from":0,"seq":38654705695,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705695,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:53.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:53.969 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:53.979 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-18T14:08:53.939108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003211"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.72199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.65200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.52200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:53.980 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:53.980 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:53.980 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:53.980 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:53.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:54.330 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:54.330 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":145,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:54.344 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:54.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:54.634 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:54.638 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:54.648 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-18T14:08:53.939108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003211"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.72199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.65200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.52200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:57.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:57.942 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:57.944 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:57.955 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":83,"stamp":"2024-09-18T14:08:55.939722+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003273"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.72199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.65200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.52200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:57.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:58.242 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:58.256 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-18T14:08:57.940322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.72199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.65200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.52200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:08:58.256 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:08:58.256 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:08:58.257 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:08:58.257 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:08:58.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:08:58.602 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:58.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":149,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:08:58.621 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:08:58.622 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:08:58.905 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:08:58.908 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:08:58.919 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":84,"stamp":"2024-09-18T14:08:57.940322+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.72199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.68000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.65200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705696,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.52200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705696,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:01.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:02.222 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:02.224 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:02.241 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-18T14:09:01.941477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.97399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67900000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.77800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:02.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:02.543 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-18T14:09:01.941477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.97399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67900000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.77800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:02.544 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:02.544 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:02.545 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:02.545 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:02.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:02.879 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:02.879 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":153,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:02.897 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:02.897 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:03.176 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:03.179 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:03.190 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":86,"stamp":"2024-09-18T14:09:01.941477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.97399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.76800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705697,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67900000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.752},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.77800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705697,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:06.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:06.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:06.496 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:06.507 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-18T14:09:05.942690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003581"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.51500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705698,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705698,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:06.507 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:06.787 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:06.790 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:06.800 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-18T14:09:05.942690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003581"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.51500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705698,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705698,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:06.800 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:06.801 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:06.801 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:06.801 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:06.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:07.152 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:07.152 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":158,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:07.166 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:07.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:07.456 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:07.459 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:07.469 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":88,"stamp":"2024-09-18T14:09:05.942690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003581"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68400000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705698,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.51500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705698,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705698,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:10.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:10.762 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:10.765 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:10.775 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-18T14:09:09.943926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003604"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.73599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.628},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.752}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.78000000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:10.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:11.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:11.061 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:11.073 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-18T14:09:09.943926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003604"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.73599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.628},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.752}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.78000000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:11.074 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:11.074 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:11.074 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:11.074 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:11.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:11.409 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:11.409 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":162,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:11.423 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:11.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:11.708 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:11.712 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:11.722 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":90,"stamp":"2024-09-18T14:09:09.943926+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003604"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.73599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705699,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.628},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.752}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.78000000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:14.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:15.026 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:15.029 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:15.039 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-18T14:09:13.945156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003679"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70499999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.505},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.71699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.623},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.67300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:15.040 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:15.326 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:15.329 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:15.340 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2024-09-18T14:09:13.945156+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003679"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70499999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.505},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.71699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.623},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.67300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:15.341 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:15.341 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:15.341 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:15.341 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:15.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:15.684 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":166,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:15.703 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:15.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:15.992 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:15.995 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:16.005 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":93,"stamp":"2024-09-18T14:09:15.945725+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003660"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70499999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.505},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.71699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.623},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.67300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:19.007 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:19.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:19.304 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:19.319 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-18T14:09:17.946241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003550"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70499999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.505},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.71699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.623},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.67300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:19.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:19.610 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:19.614 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:19.625 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":94,"stamp":"2024-09-18T14:09:17.946241+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003550"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.70499999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705700,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.505},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.71699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.52500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.623},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.67300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705700,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:19.626 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:19.627 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:19.627 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:19.627 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:19.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:19.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:19.976 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":170,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:19.994 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:19.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:20.284 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:20.287 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:20.297 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2024-09-18T14:09:19.946756+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003474"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.64700000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.74099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:23.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:23.596 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:23.600 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:23.610 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-18T14:09:21.947328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003401"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.64700000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.74099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:23.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:23.897 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:23.899 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:23.910 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":96,"stamp":"2024-09-18T14:09:21.947328+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003401"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.50900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.72499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705701,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.64700000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.74099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705701,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:23.911 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:23.911 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:23.911 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:23.911 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:23.911 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:24.254 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":175,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:24.275 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:24.276 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:24.558 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:24.561 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:24.571 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":97,"stamp":"2024-09-18T14:09:23.947907+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003280"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.90100000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.55100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.624}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.72799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:27.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:27.869 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:27.872 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:27.883 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2024-09-18T14:09:25.948378+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003221"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.90100000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.55100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.624}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.72799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:27.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:28.174 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:28.178 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:28.188 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-18T14:09:27.948868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003143"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.90100000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.55100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.624}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.72799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:28.189 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:28.189 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:28.189 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:28.189 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:28.190 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:28.538 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:28.538 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":179,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:28.552 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:28.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:28.854 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":99,"stamp":"2024-09-18T14:09:27.948868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003143"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.90100000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705702,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.55100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.624}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.72799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705702,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:31.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:32.160 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:32.163 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:32.178 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-18T14:09:31.949908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003152"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.65900000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.746},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:32.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:32.469 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:32.472 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:32.482 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-18T14:09:31.949908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003152"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.65900000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.746},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:32.483 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:32.483 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:32.483 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:32.484 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:32.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:32.831 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:32.831 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":183,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:32.845 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:32.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:33.133 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:33.136 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:33.147 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-09-18T14:09:31.949908+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003152"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.65900000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.746},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705703,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705703,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:36.150 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:36.450 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:36.453 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:36.464 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-18T14:09:35.950948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003041"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68700000000000006}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.74199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.56699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.621}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75}]}]},{"osd":1,"up_from":0,"seq":38654705704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:36.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:36.754 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:36.757 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:36.767 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-18T14:09:35.950948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003041"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68700000000000006}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.74199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.56699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.621}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75}]}]},{"osd":1,"up_from":0,"seq":38654705704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:36.768 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:36.768 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:36.768 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:36.769 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:36.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:37.111 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:37.111 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":187,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:37.124 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:37.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:37.411 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:37.414 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:37.424 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":103,"stamp":"2024-09-18T14:09:35.950948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003041"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.68700000000000006}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.74199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705704,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.56699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.621}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.75}]}]},{"osd":1,"up_from":0,"seq":38654705704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705704,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:40.426 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:40.725 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:40.738 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-18T14:09:39.952016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57299999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.81399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.748},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:40.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:41.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:41.029 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:41.040 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-18T14:09:39.952016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57299999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.81399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.748},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:41.041 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:41.041 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:41.041 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:41.041 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:41.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:41.385 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:41.385 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":192,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:41.398 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:41.399 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:41.686 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:41.689 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:41.700 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":105,"stamp":"2024-09-18T14:09:39.952016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.57299999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.81399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.68000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705705,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.748},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.63400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705705,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:44.702 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:45.008 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:45.011 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:45.022 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-18T14:09:43.953118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.82199999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:45.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:45.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:45.308 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:45.319 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":107,"stamp":"2024-09-18T14:09:43.953118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.82199999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:45.319 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:45.320 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:45.320 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:45.320 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:45.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:45.664 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:45.664 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":196,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:45.678 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:45.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:45.965 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:45.968 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:45.979 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":108,"stamp":"2024-09-18T14:09:45.953686+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003325"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.82199999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:48.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:49.290 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:49.295 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:49.308 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-18T14:09:47.954191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003243"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.82199999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:49.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:49.609 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":109,"stamp":"2024-09-18T14:09:47.954191+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003243"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.61099999999999999,"15min":0.61499999999999999},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.71899999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.626,"15min":0.622},"min":{"1min":0.504,"5min":0.433,"15min":0.433},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.68500000000000005,"15min":0.68799999999999994},"min":{"1min":0.439,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.80600000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.51600000000000001,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.77800000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:08:42 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.76600000000000001,"15min":0.76400000000000001},"min":{"1min":0.67900000000000005,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.86399999999999999,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.72899999999999998,"15min":0.73399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705706,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61499999999999999,"15min":0.61899999999999999},"min":{"1min":0.49199999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.752,"5min":0.876,"15min":0.876},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.84799999999999998,"5min":0.996,"15min":0.996},"last":0.54000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.67400000000000004,"15min":0.66800000000000004},"min":{"1min":0.57999999999999996,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.91300000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.629,"15min":0.627},"min":{"1min":0.55800000000000005,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.77400000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:08:43 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.65800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":0.85299999999999998,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.69999999999999996,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89500000000000002,"5min":1.222,"15min":1.222},"last":0.82199999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705706,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:49.610 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:49.610 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:49.611 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:49.611 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:49.611 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:49.951 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:49.951 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":200,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:49.965 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:49.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:50.260 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2024-09-18T14:09:49.954677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.625}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.69099999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:53.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:53.562 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:53.564 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:53.575 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-18T14:09:51.955168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003151"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.625}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.69099999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:53.575 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:53.860 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:53.863 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:53.874 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":111,"stamp":"2024-09-18T14:09:51.955168+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003151"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.625}]}]},{"osd":2,"up_from":9,"seq":38654705707,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.69099999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.77500000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:53.875 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:53.875 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:53.875 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:53.875 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:53.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:54.223 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:54.223 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":205,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:54.241 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:54.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:54.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:54.527 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:54.537 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2024-09-18T14:09:53.955724+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003127"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.751}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.70999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:57.539 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:57.836 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:57.839 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:57.850 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":113,"stamp":"2024-09-18T14:09:55.956145+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.751}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.70999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:57.851 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:58.146 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:58.149 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:58.159 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-18T14:09:57.956693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003006"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.751}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.70999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:09:58.160 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:09:58.160 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:09:58.160 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:09:58.161 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:09:58.161 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:09:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:58.503 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":209,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:09:58.517 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:09:58.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:09:58.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:09:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:09:58.813 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":114,"stamp":"2024-09-18T14:09:57.956693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003006"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54300000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.60499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.751}]}]},{"osd":2,"up_from":9,"seq":38654705708,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.70999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705708,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:01.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:02.114 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:02.117 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:02.128 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-18T14:10:01.957915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003237"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.73699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.64300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:02.129 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:02.417 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:02.420 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:02.430 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-18T14:10:01.957915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003237"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.73699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.64300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:02.431 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:02.431 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:02.431 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:02.431 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:02.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:02.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:02.780 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":213,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:02.793 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:02.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:03.083 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:03.086 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:03.098 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-18T14:10:01.957915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003237"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.73699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705709,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.64300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705709,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:06.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:06.416 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-18T14:10:05.958909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.83499999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705710,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705710,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:06.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:06.707 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:06.718 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-18T14:10:05.958909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.83499999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705710,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705710,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:06.718 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:06.719 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:06.719 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:06.719 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:06.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:07.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:07.067 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":217,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:07.081 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:07.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:07.365 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:07.365 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:07.376 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2024-09-18T14:10:05.958909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003184"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705710,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.48799999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.83499999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705710,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705710,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:10.378 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:10.672 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:10.675 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:10.685 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-18T14:10:09.960005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003312"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.72199999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.86799999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:10.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:10.970 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:10.972 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:10.983 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-18T14:10:09.960005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003312"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.72199999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.86799999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:10.984 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:10.984 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:10.984 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:10.984 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:10.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:11.325 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:11.325 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":222,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:11.339 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:11.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:11.625 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:11.627 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:11.640 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":120,"stamp":"2024-09-18T14:10:09.960005+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003312"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705711,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.72199999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.86799999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705711,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:14.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:14.953 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:14.955 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:14.966 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-18T14:10:13.961113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003198"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.498},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.54900000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.628}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:14.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:15.252 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:15.254 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:15.265 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-18T14:10:13.961113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003198"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.498},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.54900000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.628}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:15.266 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:15.266 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:15.266 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:15.266 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:15.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:15.609 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":226,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:15.627 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:15.628 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:15.914 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:15.931 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2024-09-18T14:10:13.961113+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003198"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.498},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.54900000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.628}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:18.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:19.232 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:19.249 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-18T14:10:17.962247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003338"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.498},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.54900000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.628}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:19.250 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:19.537 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:19.540 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:19.554 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-09-18T14:10:17.962247+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003338"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705712,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.498},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.54900000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.628}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.78600000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705712,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:19.555 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:19.555 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:19.555 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:19.555 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:19.555 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:19.902 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:19.902 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":230,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:19.916 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:19.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:20.205 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:20.218 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":125,"stamp":"2024-09-18T14:10:19.962734+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003288"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.73299999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.66200000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:23.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:23.513 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:23.516 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:23.526 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-18T14:10:21.963228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003222"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.73299999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.66200000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:23.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:23.817 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:23.820 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:23.831 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-09-18T14:10:21.963228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003222"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.73299999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.65100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705713,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.66200000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705713,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:23.832 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:23.832 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:23.832 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:23.832 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:23.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:24.178 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:24.179 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":235,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:24.191 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:24.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:24.473 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:24.476 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:24.486 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2024-09-18T14:10:23.963739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003155"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.90900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71799999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.69199999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.752}]}]},{"osd":1,"up_from":0,"seq":38654705714,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705714,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:27.488 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:27.788 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:27.791 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:27.801 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":128,"stamp":"2024-09-18T14:10:25.964366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003252"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.90900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71799999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.69199999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.752}]}]},{"osd":1,"up_from":0,"seq":38654705714,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705714,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:27.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:28.095 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:28.098 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:28.109 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-18T14:10:27.965041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003362"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.90900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71799999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.69199999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.752}]}]},{"osd":1,"up_from":0,"seq":38654705715,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705714,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:28.110 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:28.110 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:28.110 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:28.110 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:28.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:28.460 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:28.461 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":239,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:28.477 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:28.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:28.760 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:28.762 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:28.773 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2024-09-18T14:10:27.965041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003362"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.90900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71799999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705714,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.69199999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.752}]}]},{"osd":1,"up_from":0,"seq":38654705715,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705714,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:31.775 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:32.071 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:32.073 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:32.089 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-18T14:10:31.966143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003408"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.5},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.78700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.63500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.70899999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.749}]}]},{"osd":1,"up_from":0,"seq":38654705715,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705715,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:32.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:32.383 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:32.385 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:32.396 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-18T14:10:31.966143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003408"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.5},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.78700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.63500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.70899999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.749}]}]},{"osd":1,"up_from":0,"seq":38654705715,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705715,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:32.397 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:32.397 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:32.397 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:32.397 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:32.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:32.736 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:32.736 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":243,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:32.754 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:32.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:33.044 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:33.047 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:33.058 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2024-09-18T14:10:31.966143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003408"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.5},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.78700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705715,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.63500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.70899999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.749}]}]},{"osd":1,"up_from":0,"seq":38654705715,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705715,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:36.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:36.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:36.376 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-18T14:10:35.967290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.58899999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.38200000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:36.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:36.660 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:36.663 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:36.673 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-18T14:10:35.967290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.58899999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.38200000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:36.674 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:36.674 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:36.674 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:36.674 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:36.674 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:37.022 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:37.022 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":247,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:37.036 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:37.037 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:37.318 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:37.321 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:37.331 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":133,"stamp":"2024-09-18T14:10:35.967290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.58899999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.71199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705716,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.38200000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705716,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:40.333 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:40.627 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:40.630 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:40.646 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-18T14:10:39.968370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003329"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.79800000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.65500000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:40.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:40.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:40.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-18T14:10:39.968370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003329"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.79800000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.65500000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:40.949 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:40.949 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:40.949 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:40.949 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:40.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:41.306 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:41.306 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":252,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:41.320 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:41.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:41.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:41.611 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:41.626 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2024-09-18T14:10:39.968370+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003329"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.65300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.79800000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705717,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.65500000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705717,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:44.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:44.925 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:44.928 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:44.939 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-18T14:10:43.969615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003471"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54600000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.51000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.442},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59199999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:44.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:45.228 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:45.231 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:45.243 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-18T14:10:43.969615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003471"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54600000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.51000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.442},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59199999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:45.244 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:45.244 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:45.244 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:45.244 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:45.245 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:45.589 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:45.589 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":256,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:45.603 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:45.603 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:45.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:45.893 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:45.904 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-09-18T14:10:43.969615+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003471"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54600000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.51000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.442},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59199999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:48.905 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:49.202 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:49.215 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-18T14:10:47.970712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003421"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54600000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.51000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.442},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59199999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:49.216 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:49.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:49.505 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:49.519 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":139,"stamp":"2024-09-18T14:10:47.970712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003421"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60799999999999998,"15min":0.61399999999999999},"min":{"1min":0.40899999999999997,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.80100000000000005,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.626},"min":{"1min":0.57299999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54600000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69599999999999995,"15min":0.69199999999999995},"min":{"1min":0.48199999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67200000000000004,"15min":0.67500000000000004},"min":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.81999999999999995,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:09:44 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76500000000000001,"15min":0.76400000000000001},"min":{"1min":0.56999999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.72399999999999998,"15min":0.73199999999999998},"min":{"1min":0.52100000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85899999999999999,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.60499999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705718,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.60799999999999998,"15min":0.61699999999999999},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.64700000000000002,"15min":0.65300000000000002},"min":{"1min":0.46600000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82799999999999996,"5min":0.996,"15min":0.996},"last":0.51000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.85299999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.442},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63300000000000001,"15min":0.628},"min":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.59199999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:09:45 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.78200000000000003,"15min":0.79100000000000004},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.61099999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.89600000000000002,"5min":1.222,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705718,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:49.520 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:49.520 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:49.520 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:49.520 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:49.520 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:49.866 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:49.866 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":260,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:49.880 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:49.881 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:50.164 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:50.166 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:50.175 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2024-09-18T14:10:49.971235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003415"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.79600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.59499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:53.177 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:53.474 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:53.478 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:53.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-18T14:10:51.971803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003433"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.79600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.59499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:53.489 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:53.771 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:53.775 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:53.786 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2024-09-18T14:10:51.971803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003433"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.79600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705719,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.59499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705719,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:53.786 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:53.786 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:53.787 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:53.787 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:53.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:54.125 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:54.125 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":265,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:54.139 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:54.139 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:54.422 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:54.425 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:54.435 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2024-09-18T14:10:53.972324+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003374"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64600000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.39400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.503},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.65100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.84499999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705720,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705720,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:57.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:57.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:57.738 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:57.748 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":143,"stamp":"2024-09-18T14:10:55.973025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64600000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.39400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.503},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.65100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.84499999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705720,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705720,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:57.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:58.030 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:58.034 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:58.044 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-18T14:10:57.973654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003467"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64600000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.39400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.503},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.65100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.84499999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705720,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:10:58.045 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:10:58.045 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:10:58.045 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:10:58.045 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:10:58.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:10:58.390 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:58.390 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":269,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:10:58.404 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:10:58.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:10:58.690 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:10:58.692 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:10:58.703 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":144,"stamp":"2024-09-18T14:10:57.973654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003467"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64600000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705720,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.39400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.503},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.65100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.84499999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705720,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:01.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:02.004 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:02.008 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:02.017 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-18T14:11:01.974735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003499"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.746},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67300000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.628}]}]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.65000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.753}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.83499999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:02.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:02.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:02.310 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:02.321 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-18T14:11:01.974735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003499"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.746},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67300000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.628}]}]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.65000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.753}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.83499999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:02.321 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:02.321 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:02.322 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:02.322 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:02.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:02.664 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:02.664 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":273,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:02.678 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:02.678 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:02.962 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:02.965 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:02.980 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2024-09-18T14:11:01.974735+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003499"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.746},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.67300000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.53000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.628}]}]},{"osd":2,"up_from":9,"seq":38654705721,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.65000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.753}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.86699999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.83499999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705721,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:05.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:06.281 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:06.296 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-18T14:11:05.975817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003493"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.96999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":1.034}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.44700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:06.297 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:06.581 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:06.594 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-18T14:11:05.975817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003493"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.96999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":1.034}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.44700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:06.595 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:06.595 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:06.595 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:06.596 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:06.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:06.943 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:06.943 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":277,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:06.957 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:06.958 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:07.241 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:07.244 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:07.255 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":148,"stamp":"2024-09-18T14:11:05.975817+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003493"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.96999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":1.034}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.64500000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705722,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.44700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705722,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:10.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:10.559 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:10.562 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:10.572 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-18T14:11:09.976918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003264"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.92700000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.83799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.72299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.89900000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705723,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705723,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:10.573 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:10.860 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:10.863 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:10.873 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-18T14:11:09.976918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003264"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.92700000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.83799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.72299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.89900000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705723,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705723,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:10.874 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:10.874 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:10.874 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:10.874 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:10.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:11.226 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:11.226 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":282,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:11.240 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:11.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:11.523 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:11.536 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2024-09-18T14:11:09.976918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003264"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.754}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.92700000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705723,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.83799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.72299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.89900000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705723,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705723,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:14.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:14.844 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:14.847 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:14.858 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-18T14:11:13.978044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:14.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:15.150 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:15.153 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:15.164 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-18T14:11:13.978044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:15.165 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:15.165 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:15.165 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:15.165 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:15.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:15.508 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:15.581 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":286,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:15.583 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:15.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:15.873 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:15.876 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:15.887 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-18T14:11:13.978044+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:18.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:19.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:19.198 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:19.209 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-18T14:11:17.979141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003323"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:19.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:19.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:19.503 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:19.514 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-18T14:11:17.979141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003323"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705724,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705724,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:19.515 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:19.515 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:19.515 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:19.515 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:19.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:19.860 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:19.861 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":290,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:19.874 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:19.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:20.163 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:20.177 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":155,"stamp":"2024-09-18T14:11:19.979622+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.62}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.86399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:23.178 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:23.475 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:23.478 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:23.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-18T14:11:21.980003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.62}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.86399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:23.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:23.771 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:23.773 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:23.784 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":156,"stamp":"2024-09-18T14:11:21.980003+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003084"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.57199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.62}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.86399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705725,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85499999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705725,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:23.784 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:23.785 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:23.785 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:23.785 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:23.785 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:24.121 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:24.122 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":295,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:24.135 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:24.135 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:24.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:24.435 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:24.446 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-09-18T14:11:23.980608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.98199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.86699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:27.448 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:27.747 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:27.750 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:27.761 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-18T14:11:25.981196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003152"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.98199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.86699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:27.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:28.040 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:28.044 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:28.054 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-18T14:11:27.981822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003197"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.98199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.86699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:28.055 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:28.055 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:28.055 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:28.055 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:28.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:28.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:28.398 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":299,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:28.413 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:28.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:28.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:28.699 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:28.710 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-18T14:11:27.981822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003197"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.98199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705726,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.86699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705726,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:31.711 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:32.014 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:32.017 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:32.027 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-18T14:11:31.982840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003218"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.98199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85699999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:32.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:32.319 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:32.322 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:32.337 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-18T14:11:31.982840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003218"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.98199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85699999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:32.337 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:32.338 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:32.338 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:32.338 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:32.338 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:32.683 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:32.684 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":303,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:32.697 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:32.697 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:32.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:32.990 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:33.000 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-18T14:11:31.982840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003218"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.98199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705727,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.85699999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705727,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:36.002 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:36.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:36.305 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:36.316 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-18T14:11:35.983846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003237"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.755}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.65100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:36.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:36.604 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:36.618 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-18T14:11:35.983846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003237"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.755}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.65100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:36.619 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:36.619 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:36.619 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:36.619 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:36.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:36.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:36.969 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":307,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:36.982 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:36.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:37.276 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:37.279 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:37.290 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2024-09-18T14:11:35.983846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003237"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.69299999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.755}]}]},{"osd":2,"up_from":9,"seq":38654705728,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.46700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.60599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.65100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705728,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:40.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:40.595 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:40.598 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:40.609 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-18T14:11:39.984989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69799999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.66400000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.76500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.83299999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705729,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705729,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:40.609 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:40.898 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:40.901 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:40.912 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-18T14:11:39.984989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69799999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.66400000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.76500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.83299999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705729,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705729,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:40.913 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:40.913 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:40.913 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:40.913 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:40.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:41.253 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:41.253 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":312,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:41.266 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:41.267 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:41.552 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:41.555 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:41.565 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-18T14:11:39.984989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.69799999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705729,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.66400000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.76500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.83299999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705729,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705729,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:44.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:44.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:44.881 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-18T14:11:43.986165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003325"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.62},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.64000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:44.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:45.170 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:45.174 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:45.184 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-18T14:11:43.986165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003325"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.62},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.64000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:45.185 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:45.185 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:45.185 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:45.185 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:45.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:45.534 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:45.535 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":316,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:45.554 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:45.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:45.843 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:45.861 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":167,"stamp":"2024-09-18T14:11:43.986165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003325"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.62},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.64000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:48.863 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:49.162 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:49.165 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:49.179 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-18T14:11:47.987409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003562"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.62},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.64000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:49.180 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:11:49.492+0000 7fb8de59c700 0 --2- 172.21.15.16:0/279435477 >> [v2:172.21.15.16:3302/0,v1:172.21.15.16:6791/0] conn(0x7fb8e00697b0 0x7fb8e01207f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:11:49.674 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:49.678 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:49.688 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2024-09-18T14:11:47.987409+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003562"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56200000000000006}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.62},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705730,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59599999999999997,"15min":0.61299999999999999},"min":{"1min":0.45400000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69999999999999996,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.63500000000000001,"15min":0.64900000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.70899999999999996,"5min":0.996,"15min":0.996},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.442,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.81999999999999995,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.63800000000000001,"15min":0.63},"min":{"1min":0.51300000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.84299999999999997,"5min":0.87,"15min":0.87},"last":0.64000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77800000000000002,"15min":0.79000000000000004},"min":{"1min":0.67700000000000005,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.90500000000000003,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.76800000000000002,"15min":0.76500000000000001},"min":{"1min":0.66200000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.86799999999999999,"5min":1.222,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705730,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:49.689 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:49.689 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:49.689 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:49.689 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:49.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:50.029 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:50.029 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":320,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:50.043 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:50.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:50.332 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:50.335 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:50.345 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":170,"stamp":"2024-09-18T14:11:49.987925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003427"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.58699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.71299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.86899999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:53.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:53.639 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:53.642 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:53.652 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-18T14:11:51.988470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.58699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.71299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.86899999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:53.653 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:53.950 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:53.953 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:53.964 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":171,"stamp":"2024-09-18T14:11:51.988470+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003481"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.59499999999999997,"15min":0.60999999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.77000000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63800000000000001,"15min":0.626},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.85999999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.58699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69199999999999995},"min":{"1min":0.51900000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86399999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.67300000000000004,"15min":0.67500000000000004},"min":{"1min":0.57899999999999996,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:10:45 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.76100000000000001,"15min":0.76300000000000001},"min":{"1min":0.60199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.82399999999999995,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.71099999999999997,"15min":0.72799999999999998},"min":{"1min":0.52000000000000002,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.78500000000000003,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.63800000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705731,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.71299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.86899999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705731,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:53.965 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:53.965 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:53.965 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:53.965 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:53.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:54.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:54.307 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":325,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:54.321 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:54.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:54.624 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2024-09-18T14:11:53.988969+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003396"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.498},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.628},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.75},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:57.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:57.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:57.936 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:57.946 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2024-09-18T14:11:55.989530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003364"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.498},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.628},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.75},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:57.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:58.241 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:58.259 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-18T14:11:57.990034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003163"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.498},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.628},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.75},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:11:58.260 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:11:58.260 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:11:58.260 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:11:58.260 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:11:58.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:11:58.617 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:58.617 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":329,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:11:58.631 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:11:58.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:11:58.921 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:11:58.925 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:11:58.935 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":174,"stamp":"2024-09-18T14:11:57.990034+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003163"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.498},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.628},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.82099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705732,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.75},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705732,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:01.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:02.238 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:02.242 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:02.252 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-18T14:12:01.991098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003172"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.75}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.96699999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705733,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:02.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:02.536 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:02.539 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:02.550 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-18T14:12:01.991098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003172"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.75}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.96699999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705733,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:02.550 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:02.551 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:02.551 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:02.551 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:02.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:02.894 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:02.895 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":333,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:02.914 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:02.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:03.202 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:03.215 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":176,"stamp":"2024-09-18T14:12:01.991098+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003172"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70299999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.75}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705733,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.96699999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705733,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:06.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:06.523 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:06.527 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:06.539 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-18T14:12:05.992419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003449"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.626}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66500000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.96699999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705735,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:06.540 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:06.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:06.829 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:06.840 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-18T14:12:05.992419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003449"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.626}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66500000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.96699999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705735,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:06.841 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:06.841 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:06.841 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:06.841 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:06.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:07.189 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:07.189 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":338,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:07.203 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:07.203 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:07.488 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:07.492 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:07.505 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":178,"stamp":"2024-09-18T14:12:05.992419+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003449"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.626}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":1.1619999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66500000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705734,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.96699999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705735,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705734,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:10.507 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:10.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:10.806 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:10.816 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-18T14:12:09.993603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003569"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.47799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.54300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.622}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.72199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705735,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:10.817 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:11.104 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:11.107 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:11.117 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-18T14:12:09.993603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003569"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.47799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.54300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.622}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.72199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705735,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:11.118 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:11.118 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:11.118 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:11.118 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:11.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:11.465 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:11.465 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":342,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:11.480 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:11.480 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:11.766 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:11.769 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:11.780 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":180,"stamp":"2024-09-18T14:12:09.993603+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003569"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.47799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.621}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.54300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705735,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.40200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.622}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.72199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705735,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:14.781 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:15.091 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:15.094 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:15.109 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-18T14:12:13.994753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.67000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.64600000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705737,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:15.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:15.395 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:15.408 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2024-09-18T14:12:13.994753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003655"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.67000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.64600000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705737,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:15.408 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:15.409 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:15.409 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:15.409 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:15.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:15.752 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:15.753 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":346,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:15.771 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:15.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:16.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:16.066 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:16.076 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2024-09-18T14:12:15.995450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003753"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.67000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.64600000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705737,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:19.078 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:19.379 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:19.383 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:19.397 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-18T14:12:17.996064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003645"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.67000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.64600000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:19.398 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:19.690 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:19.693 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:19.704 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":184,"stamp":"2024-09-18T14:12:17.996064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003645"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.81799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705736,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.67000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.64600000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705736,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:19.704 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:19.705 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:19.705 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:19.705 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:19.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:20.047 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:20.048 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":350,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:20.061 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:20.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:20.357 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:20.360 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:20.375 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":185,"stamp":"2024-09-18T14:12:19.996584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003522"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.63500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705737,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:23.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:23.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:23.683 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:23.693 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-18T14:12:21.997171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003568"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.63500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705737,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:23.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:23.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:23.980 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:23.990 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":186,"stamp":"2024-09-18T14:12:21.997171+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003568"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705737,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.63500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705737,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:23.991 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:23.991 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:23.992 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:23.992 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:23.992 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:24.344 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:24.345 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":355,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:24.358 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:24.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:24.649 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:24.652 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:24.663 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":187,"stamp":"2024-09-18T14:12:23.997696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.89800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78100000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705739,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:27.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:27.962 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:27.965 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:27.976 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":188,"stamp":"2024-09-18T14:12:25.998412+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003659"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.89800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78100000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705739,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:27.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:28.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:28.264 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:28.280 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-18T14:12:27.998914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.89800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78100000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:28.281 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:28.281 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:28.281 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:28.281 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:28.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:28.622 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:28.622 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":359,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:28.636 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:28.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:28.922 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:28.925 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:28.937 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":189,"stamp":"2024-09-18T14:12:27.998914+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.76800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705738,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.89800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78100000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705738,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:31.938 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:32.236 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:32.239 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:32.250 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-18T14:12:31.999931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003347"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.627},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.85099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.89000000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705739,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:32.251 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:32.534 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:32.537 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:32.548 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-18T14:12:31.999931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003347"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.627},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.85099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.89000000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705739,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:32.549 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:32.549 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:32.549 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:32.549 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:32.549 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:32.896 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:32.896 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":363,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:32.910 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:32.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:33.194 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:33.197 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:33.208 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-18T14:12:31.999931+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003347"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705739,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.627},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.85099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.89000000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705739,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:36.210 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:36.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:36.510 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:36.525 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-18T14:12:36.001000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003304"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.73099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.88},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.83599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.47099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":1.107},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.80500000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705741,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:36.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:36.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:36.812 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:36.823 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-18T14:12:36.001000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003304"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.73099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.88},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.83599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.47099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":1.107},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.80500000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705741,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:36.823 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:36.824 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:36.824 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:36.824 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:36.824 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:37.175 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:37.175 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":368,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:37.189 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:37.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:37.471 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:37.473 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:37.484 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":193,"stamp":"2024-09-18T14:12:36.001000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003304"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.73099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.88},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.83599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705740,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.47099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":1.107},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.80500000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705741,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705740,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:40.485 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:40.781 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:40.783 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:40.794 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-18T14:12:40.002078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70699999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.624}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705741,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:40.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:41.085 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:41.088 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:41.099 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-18T14:12:40.002078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70699999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.624}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705741,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:41.100 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:41.100 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:41.100 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:41.100 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:41.100 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:41.448 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:41.448 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":372,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:41.461 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:41.462 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:41.754 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:41.757 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:41.767 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-18T14:12:40.002078+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003164"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70699999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.58399999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705741,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.624}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705741,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:44.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:45.071 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:45.074 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:45.084 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-18T14:12:44.003321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003389"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.749}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.78900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.624}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.755}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78300000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705743,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:45.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:45.370 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:45.373 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:45.384 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-09-18T14:12:44.003321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003389"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.749}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.78900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.624}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.755}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78300000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705743,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:45.385 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:45.385 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:45.385 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:45.385 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:45.385 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:45.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:45.735 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":376,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:45.749 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:45.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:46.036 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:46.039 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:46.054 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-09-18T14:12:46.003898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003477"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.749}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.78900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.624}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.755}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78300000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705743,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:49.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:49.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:49.355 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:49.365 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-18T14:12:48.004485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003484"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.749}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.78900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.624}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.755}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78300000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:49.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:49.654 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:49.658 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:49.668 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-18T14:12:48.004485+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003484"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.749}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.78900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705742,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.58699999999999997,"15min":0.60999999999999999},"min":{"1min":0.38800000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.84599999999999997,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.64600000000000002},"min":{"1min":0.39400000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.83799999999999997,"5min":0.996,"15min":0.996},"last":0.624}]},{"osd":1,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.65600000000000003,"15min":0.66200000000000003},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74299999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.502,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.82599999999999996,"5min":0.87,"15min":0.87},"last":0.755}]},{"osd":3,"last update":"Wed Sep 18 14:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.77900000000000003,"15min":0.79000000000000004},"min":{"1min":0.66800000000000004,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.92100000000000004,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.77200000000000002,"15min":0.76600000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.93200000000000005,"5min":1.222,"15min":1.222},"last":0.78300000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705742,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:49.669 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:49.669 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:49.669 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:49.669 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:49.669 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:50.015 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:50.015 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":380,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:50.026 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:50.027 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:50.320 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:50.323 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:50.334 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":200,"stamp":"2024-09-18T14:12:50.005101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003547"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.376}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.63400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.745}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.69299999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705743,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:53.335 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:53.626 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:53.630 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:53.640 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-18T14:12:52.005655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003576"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.376}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.63400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.745}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.69299999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705743,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:53.641 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:53.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:53.944 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-18T14:12:52.005655+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003576"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.60599999999999998,"15min":0.61399999999999999},"min":{"1min":0.46999999999999997,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64400000000000002,"15min":0.628},"min":{"1min":0.54600000000000004,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.96999999999999997,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.69999999999999996,"15min":0.69299999999999995},"min":{"1min":0.55400000000000005,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.84999999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.52100000000000002,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.376}]},{"osd":2,"last update":"Wed Sep 18 14:11:48 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.75700000000000001,"15min":0.76100000000000001},"min":{"1min":0.61099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.71499999999999997,"15min":0.72899999999999998},"min":{"1min":0.628,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705743,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.63400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.745}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.69299999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705743,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:53.945 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:53.945 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:53.945 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:53.945 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:53.945 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:54.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:54.301 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":385,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":4,"modified":"2024-09-18T14:07:39.918946+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:54.314 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:54.314 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:54.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:54.612 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:54.623 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":202,"stamp":"2024-09-18T14:12:54.006158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003482"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.72199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705745,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:57.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:57.927 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:57.930 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:57.940 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":203,"stamp":"2024-09-18T14:12:56.006706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.72199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705745,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:57.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:58.227 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:58.230 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:58.241 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-18T14:12:58.007229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.72199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:12:58.241 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:12:58.241 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:12:58.242 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:12:58.242 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:12:58.242 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:12:58.585 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:58.585 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":389,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:12:58.599 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:12:58.599 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:12:58.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:12:58.890 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:12:58.901 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2024-09-18T14:12:58.007229+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.88500000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705744,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.46300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.72199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705744,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:01.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:02.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:02.200 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:02.211 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-18T14:13:02.008361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.80900000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67800000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.61499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705745,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:02.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:02.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:02.502 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:02.513 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-18T14:13:02.008361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.80900000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67800000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.61499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705745,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:02.514 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:02.515 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:02.515 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:02.515 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:02.515 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:02.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:02.859 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":393,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:02.874 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:02.874 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:03.156 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:03.159 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:03.174 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":206,"stamp":"2024-09-18T14:13:02.008361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003259"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.80900000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67800000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705745,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.61499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705745,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:06.176 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:06.474 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:06.477 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:06.492 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-18T14:13:06.009442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.505}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.85599999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705747,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:06.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:06.780 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:06.783 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:06.794 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-18T14:13:06.009442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.505}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.85599999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705747,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:06.794 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:06.795 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:06.795 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:06.795 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:06.795 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:07.144 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:07.145 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":398,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:07.159 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:07.159 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:07.445 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:07.448 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:07.459 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":208,"stamp":"2024-09-18T14:13:06.009442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.505}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67300000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705746,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.59099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.85599999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705747,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705746,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:10.461 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:10.763 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:10.776 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-18T14:13:10.010505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003276"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.40000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.52200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.68200000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705748,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705747,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:10.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:11.064 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:11.067 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:11.078 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-18T14:13:10.010505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003276"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.40000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.52200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.68200000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705748,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705747,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:11.079 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:11.079 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:11.079 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:11.079 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:11.079 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:11.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:11.431 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":402,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:11.444 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:11.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:11.725 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:11.738 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":210,"stamp":"2024-09-18T14:13:10.010505+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003276"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.40000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.55100000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.52200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705747,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.68200000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705748,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705747,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:14.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:15.039 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:15.042 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:15.052 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-18T14:13:14.011717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003356"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.65000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.79100000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705748,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:15.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:15.341 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:15.345 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:15.355 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-18T14:13:14.011717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003356"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.65000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.79100000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705748,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:15.356 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:15.356 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:15.356 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:15.356 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:15.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:15.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:15.694 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":406,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:15.707 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:15.708 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:15.999 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:16.002 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:16.012 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":212,"stamp":"2024-09-18T14:13:14.011717+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003356"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.65000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.79100000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705748,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:19.014 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:19.311 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:19.314 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:19.325 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-18T14:13:18.012741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003298"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58299999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.79100000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705750,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:19.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:19.615 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:19.618 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:19.628 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":214,"stamp":"2024-09-18T14:13:18.012741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003298"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58299999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705748,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.79100000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705750,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:19.629 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:19.629 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:19.629 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:19.629 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:19.630 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:19.978 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:19.979 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":410,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:19.993 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:19.993 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:20.280 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:20.297 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2024-09-18T14:13:20.013296+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003384"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58299999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.76800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705750,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:23.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:23.591 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:23.594 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:23.605 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-18T14:13:22.013861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003355"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58299999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.76800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705750,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:23.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:23.898 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:23.917 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":216,"stamp":"2024-09-18T14:13:22.013861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003355"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.71099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58299999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705749,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.76800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705750,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705749,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:23.918 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:23.918 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:23.918 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:23.918 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:23.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:24.272 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:24.272 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":415,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:24.285 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:24.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:24.567 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:24.572 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:24.580 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":217,"stamp":"2024-09-18T14:13:24.014361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003169"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.59299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.76800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:27.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:27.882 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:27.885 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:27.894 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":218,"stamp":"2024-09-18T14:13:26.014738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003021"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.59299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705750,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.76800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:27.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:28.177 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:28.191 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-18T14:13:28.015199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002956"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.70799999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:28.191 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:28.192 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:28.192 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:28.192 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:28.192 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:28.540 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:28.541 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":419,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:28.554 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:28.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:28.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:28.985 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":219,"stamp":"2024-09-18T14:13:28.015199+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002956"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.70799999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705751,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:31.987 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:32.290 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:32.293 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:32.304 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-18T14:13:32.016211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.70799999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:32.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:32.586 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:32.589 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:32.599 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-18T14:13:32.016211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.70799999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:32.600 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:32.600 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:32.600 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:32.601 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:32.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:32.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:32.949 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":423,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:32.963 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:32.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:33.254 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:33.257 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:33.267 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":221,"stamp":"2024-09-18T14:13:32.016211+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002914"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.95599999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.70799999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705751,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705752,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:36.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:36.568 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:36.572 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:36.583 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-18T14:13:36.017361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.88100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.52900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705753,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705753,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:36.584 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:36.867 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:36.869 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:36.886 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-18T14:13:36.017361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.88100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.52900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705753,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705753,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:36.887 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:36.887 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:36.887 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:36.887 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:36.888 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:37.242 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:37.242 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":428,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:37.261 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:37.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:37.543 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:37.545 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:37.556 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":223,"stamp":"2024-09-18T14:13:36.017361+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002999"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.88100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.52900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705752,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.60299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705753,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705753,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:40.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:40.857 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:40.859 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:40.870 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-18T14:13:40.018507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.83899999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.79300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.877},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.68400000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:40.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:41.158 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:41.161 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:41.172 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-18T14:13:40.018507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.83899999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.79300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.877},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.68400000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:41.173 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:41.173 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:41.173 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:41.173 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:41.173 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:41.515 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:41.515 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":432,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:41.529 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:41.529 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:41.818 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:41.821 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:41.832 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2024-09-18T14:13:40.018507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003307"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.83899999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705753,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.63600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.79300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.877},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.68400000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705754,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:44.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:45.134 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:45.137 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:45.147 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-18T14:13:44.019831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003620"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.71199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.66600000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.747},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55200000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:45.148 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:45.437 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:45.439 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:45.450 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":227,"stamp":"2024-09-18T14:13:44.019831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003620"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.71199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.66600000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.747},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55200000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:45.451 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:45.451 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:45.451 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:45.451 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:45.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:45.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:45.800 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":436,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:45.813 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:45.814 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:46.105 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:46.108 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:46.119 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-18T14:13:46.020489+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003707"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.55200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.71199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64300000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705754,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.66600000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.747},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.55200000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705755,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:49.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:49.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:49.429 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:49.439 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-18T14:13:48.020893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003532"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.57099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.69099999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:49.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:49.734 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:49.737 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:49.747 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":229,"stamp":"2024-09-18T14:13:48.020893+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003532"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.57099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.69099999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:49.748 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:49.748 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:49.748 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:49.748 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:49.749 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:50.093 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:50.093 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":440,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:50.107 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:50.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:50.403 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:50.406 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:50.417 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":230,"stamp":"2024-09-18T14:13:50.021443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003528"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.57099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.69099999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:53.418 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:53.716 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:53.730 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-18T14:13:52.021959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.57099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.69099999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:53.731 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:54.024 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:54.027 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:54.037 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":231,"stamp":"2024-09-18T14:13:52.021959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.59899999999999998,"15min":0.61099999999999999},"min":{"1min":0.41999999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.65000000000000002,"15min":0.63},"min":{"1min":0.47799999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81399999999999995,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.69899999999999995,"15min":0.69299999999999995},"min":{"1min":0.50700000000000001,"5min":0.439,"15min":0.40200000000000002},"max":{"1min":0.98099999999999998,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.67800000000000005,"15min":0.67700000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92200000000000004,"5min":1.034,"15min":1.034},"last":0.57099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:12:48 2024","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.75900000000000001,"15min":0.76200000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.72799999999999998,"15min":0.73399999999999999},"min":{"1min":0.49399999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":3.0720000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705755,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.58599999999999997,"15min":0.60999999999999999},"min":{"1min":0.40500000000000003,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86399999999999999,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.622,"15min":0.64500000000000002},"min":{"1min":0.40200000000000002,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.996},"last":0.51400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.66800000000000004,"15min":0.66600000000000004},"min":{"1min":0.502,"5min":0.372,"15min":0.372},"max":{"1min":1.107,"5min":1.107,"15min":1.107},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65500000000000003,"15min":0.63500000000000001},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:12:47 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.77600000000000002,"15min":0.78900000000000003},"min":{"1min":0.65100000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.64600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.222},"last":0.69099999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705756,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:54.038 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:54.038 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:54.038 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:54.038 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:54.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:54.387 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":445,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:54.406 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:54.406 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:54.689 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:54.692 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:54.703 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":232,"stamp":"2024-09-18T14:13:54.022500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003334"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.433},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.65900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.82899999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705757,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705757,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:57.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:58.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:58.016 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:58.031 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":233,"stamp":"2024-09-18T14:13:56.023081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003250"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.433},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.65900000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705756,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.82899999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705757,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705757,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:58.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:58.316 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:58.321 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:58.337 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-18T14:13:58.023598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003108"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.625}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.871}]}]},{"osd":1,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:13:58.338 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:13:58.338 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:13:58.338 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:13:58.338 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:13:58.339 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:13:58.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:58.679 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":449,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:13:58.696 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:13:58.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:13:58.982 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:13:58.985 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:13:58.995 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":234,"stamp":"2024-09-18T14:13:58.023598+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003108"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.625}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.871}]}]},{"osd":1,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:01.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:02.296 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:02.299 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:02.315 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-18T14:14:02.024810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003366"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.625}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.871}]}]},{"osd":1,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:02.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:02.602 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:02.615 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-18T14:14:02.024810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003366"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.625}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.871}]}]},{"osd":1,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:02.616 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:02.616 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:02.616 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:02.616 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:02.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:02.964 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:02.964 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":453,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:02.978 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:02.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:03.266 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:03.269 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:03.280 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":236,"stamp":"2024-09-18T14:14:02.024810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003366"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.75},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61199999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.88200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705757,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.625}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68000000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.871}]}]},{"osd":1,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705758,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:06.282 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:06.582 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:06.584 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:06.595 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-18T14:14:06.025848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003347"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.52600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.54900000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.96299999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705759,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705759,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:06.596 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:06.881 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:06.885 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:06.895 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-18T14:14:06.025848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003347"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.52600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.54900000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.96299999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705759,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705759,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:06.896 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:06.896 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:06.896 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:06.896 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:06.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:07.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:07.243 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":458,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:07.257 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:07.257 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:07.542 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:07.544 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:07.556 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":238,"stamp":"2024-09-18T14:14:06.025848+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003347"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.46600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.52600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.79900000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705758,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.54900000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.96299999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705759,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705759,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:10.557 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:10.858 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:10.862 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:10.871 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-18T14:14:10.027002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003404"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.85799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.76800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705760,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705760,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:10.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:11.167 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:11.170 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:11.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-18T14:14:10.027002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003404"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.85799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.76800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705760,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705760,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:11.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:11.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:11.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:11.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:11.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:11.525 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:11.525 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":462,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:11.543 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:11.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:11.829 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:11.832 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:11.843 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":240,"stamp":"2024-09-18T14:14:10.027002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003404"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.85799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705759,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68200000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.76800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705760,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705760,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:14.844 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:15.141 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:15.144 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:15.155 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-18T14:14:14.028093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.69599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.47099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.83899999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:15.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:15.451 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:15.453 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:15.464 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":242,"stamp":"2024-09-18T14:14:14.028093+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.69599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.47099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.83899999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:15.465 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:15.465 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:15.465 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:15.465 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:15.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:15.816 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:15.817 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":466,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:15.831 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:15.831 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:16.120 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:16.123 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:16.134 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2024-09-18T14:14:16.028777+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003519"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64000000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.69599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.76300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705760,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.47099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.83899999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705761,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:19.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:19.440 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:19.444 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:19.454 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-18T14:14:18.029291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003443"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.82699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.497}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.749}]}]},{"osd":1,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:19.456 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:19.741 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:19.744 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:19.754 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":244,"stamp":"2024-09-18T14:14:18.029291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003443"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.82699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.497}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.749}]}]},{"osd":1,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:19.755 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:19.755 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:19.755 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:19.756 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:19.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:20.104 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:20.104 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":470,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2024-09-18T14:12:58.007276+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:20.118 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:20.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:20.402 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:20.405 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:20.416 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2024-09-18T14:14:20.029948+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003546"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.82699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.497}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.749}]}]},{"osd":1,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:23.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:23.719 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:23.722 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:23.733 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-18T14:14:22.030455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.82699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.497}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.749}]}]},{"osd":1,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:23.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:24.021 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:24.024 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:24.035 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2024-09-18T14:14:22.030455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.82699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705761,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.497}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.749}]}]},{"osd":1,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705762,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:24.036 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:24.036 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:24.036 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:24.036 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:24.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:24.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:24.381 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":475,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:24.400 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:24.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:24.688 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:24.703 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":247,"stamp":"2024-09-18T14:14:24.031001+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003467"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.629}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73099999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705763,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705763,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:27.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:28.004 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:28.007 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:28.018 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-18T14:14:26.031611+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003518"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705762,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705762,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.67700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.629}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73099999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705763,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705763,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:28.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:28.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:28.305 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:28.316 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-18T14:14:28.032213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003436"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.752},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.875},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.71699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.495},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66000000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.71799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:28.317 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:28.317 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:28.317 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:28.317 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:28.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:28.663 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:28.663 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":479,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:28.676 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:28.676 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:28.956 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:28.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:28.970 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":249,"stamp":"2024-09-18T14:14:28.032213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003436"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.752},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.875},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.71699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.495},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66000000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.71799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:31.972 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:32.271 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:32.274 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:32.284 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-18T14:14:32.033190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003242"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.752},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.875},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.71699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.495},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66000000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.71799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:32.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:32.577 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:32.580 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:32.591 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-18T14:14:32.033190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003242"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.752},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.875},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.71699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.495},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66000000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.71799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:32.592 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:32.592 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:32.592 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:32.592 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:32.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:32.948 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:32.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":483,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:32.962 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:32.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:33.252 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:33.255 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:33.266 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":251,"stamp":"2024-09-18T14:14:32.033190+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003242"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.64100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.752},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.875},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.71699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705763,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.495},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.61499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66000000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.71799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705764,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:36.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:36.564 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:36.568 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:36.578 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-18T14:14:36.034276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003274"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.65600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.76500000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:36.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:36.857 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:36.871 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-18T14:14:36.034276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003274"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.65600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.76500000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:36.871 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:36.872 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:36.872 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:36.872 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:36.872 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:37.221 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:37.221 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":488,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:37.239 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:37.239 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:37.527 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:37.531 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:37.541 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-18T14:14:36.034276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003274"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705764,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.65600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.76500000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705765,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:40.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:40.848 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:40.851 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:40.861 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-18T14:14:40.035346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.73699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705766,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705766,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:40.862 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:41.154 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:41.157 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:41.168 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-18T14:14:40.035346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.73699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705766,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705766,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:41.169 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:41.169 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:41.169 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:41.169 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:41.170 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:41.515 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:41.516 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":492,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:41.534 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:41.534 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:41.822 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:41.825 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:41.836 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":255,"stamp":"2024-09-18T14:14:40.035346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.50700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.66600000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705765,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.73699999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.89900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705766,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705766,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:44.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:45.138 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:45.140 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:45.150 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-18T14:14:44.036442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003251"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705767,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705767,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:45.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:45.442 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:45.445 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:45.455 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":257,"stamp":"2024-09-18T14:14:44.036442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003251"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705767,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705767,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:45.456 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:45.456 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:45.456 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:45.456 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:45.457 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:45.802 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:45.802 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":496,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:45.816 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:45.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:46.107 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:46.111 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:46.121 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":258,"stamp":"2024-09-18T14:14:46.037104+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003436"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705766,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705767,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705767,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:49.122 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:49.422 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:49.436 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-18T14:14:48.037587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003311"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.71499999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.56999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:49.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:49.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:49.735 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:49.746 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":259,"stamp":"2024-09-18T14:14:48.037587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003311"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.71499999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.56999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:49.746 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:49.747 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:49.747 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:49.747 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:49.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:50.096 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:50.097 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":500,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:50.110 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:50.111 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:50.399 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:50.402 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:50.412 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":260,"stamp":"2024-09-18T14:14:50.038183+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003382"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.71499999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.56999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:53.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:53.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:53.716 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:53.727 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-18T14:14:52.038806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003460"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.71499999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.56999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:53.728 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:54.017 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:54.020 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:54.030 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":261,"stamp":"2024-09-18T14:14:52.038806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003460"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60099999999999998,"15min":0.60999999999999999},"min":{"1min":0.38200000000000001,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.84599999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64900000000000002,"15min":0.63100000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88500000000000001,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.69799999999999995,"15min":0.69099999999999995},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.67700000000000005,"15min":0.67600000000000005},"min":{"1min":0.42799999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.88100000000000001,"5min":1.034,"15min":1.034},"last":0.71499999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.75,"15min":0.76000000000000001},"min":{"1min":0.45100000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0089999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.71899999999999997,"15min":0.72799999999999998},"min":{"1min":0.52200000000000002,"5min":0.49399999999999999,"15min":0.41199999999999998},"max":{"1min":0.83899999999999997,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.56999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705767,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.60699999999999998},"min":{"1min":0.42099999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72699999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.61199999999999999,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.73299999999999998,"5min":0.88500000000000001,"15min":0.996},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65600000000000003,"15min":0.66500000000000004},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82499999999999996,"5min":1.107,"15min":1.107},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.65300000000000002,"15min":0.63600000000000001},"min":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.79300000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:13:49 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.76400000000000001,"15min":0.78400000000000003},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":0.88500000000000001,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.78300000000000003,"15min":0.77000000000000002},"min":{"1min":0.442,"5min":0.438,"15min":0.438},"max":{"1min":0.95999999999999996,"5min":1.1359999999999999,"15min":1.222},"last":0.73399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705768,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:54.031 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:54.031 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:54.031 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:54.032 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:54.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:54.379 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:54.379 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":505,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:54.392 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:54.392 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:54.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:54.681 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:54.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":262,"stamp":"2024-09-18T14:14:54.039313+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003443"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.44400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.624}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705769,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705769,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:57.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:57.994 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:57.997 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:58.007 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-18T14:14:56.040030+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003588"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.44400000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.57599999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705768,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.624}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705769,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705769,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:58.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:58.295 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:58.298 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:58.309 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-18T14:14:58.040632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003527"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.496},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58899999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68500000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.80000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:14:58.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:14:58.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:14:58.310 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:14:58.310 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:14:58.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:14:58.651 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:58.651 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":509,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:14:58.664 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:14:58.664 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:14:58.951 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:14:58.953 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:14:58.964 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":264,"stamp":"2024-09-18T14:14:58.040632+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003527"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.496},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58899999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68500000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.80000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:01.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:02.267 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:02.271 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:02.281 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-18T14:15:02.041607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003424"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.496},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58899999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68500000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.80000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:02.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:02.575 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:02.578 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:02.594 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-18T14:15:02.041607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003424"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.496},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58899999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68500000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.80000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:02.595 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:02.595 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:02.595 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:02.595 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:02.595 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:02.942 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:02.942 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":513,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:02.956 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:02.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:03.234 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:03.236 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:03.246 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2024-09-18T14:15:02.041607+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003424"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.496},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.627}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.58899999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705769,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68500000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.80000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705770,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:06.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:06.546 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:06.548 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:06.559 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-18T14:15:06.042685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003372"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.68100000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.81699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.77400000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705771,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705771,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:06.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:06.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:06.844 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:06.854 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-18T14:15:06.042685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003372"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.68100000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.81699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.77400000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705771,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705771,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:06.855 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:06.856 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:06.856 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:06.856 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:06.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:07.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:07.199 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":518,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:07.213 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:07.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:07.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:07.496 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:07.508 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":268,"stamp":"2024-09-18T14:15:06.042685+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003372"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.68100000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.749}]}]},{"osd":2,"up_from":9,"seq":38654705770,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.81699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.77400000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705771,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705771,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:10.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:10.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:10.814 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:10.828 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-18T14:15:10.043789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.66500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.52300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68799999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.78400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:10.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:11.114 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:11.128 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-18T14:15:10.043789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.66500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.52300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68799999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.78400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:11.129 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:11.130 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:11.130 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:11.130 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:11.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:11.479 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:11.479 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":522,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:11.493 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:11.493 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:11.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:11.781 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:11.791 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2024-09-18T14:15:10.043789+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.66500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705771,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.52300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68799999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.78400000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:14.793 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:15.098 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:15.102 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:15.112 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-18T14:15:14.044841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003234"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70699999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.80300000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.52300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.77200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":1.008}]}]},{"osd":1,"up_from":0,"seq":38654705773,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705773,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:15.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:15.401 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:15.404 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:15.415 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":272,"stamp":"2024-09-18T14:15:14.044841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003234"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70699999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.80300000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.52300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.77200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":1.008}]}]},{"osd":1,"up_from":0,"seq":38654705773,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705773,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:15.416 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:15.416 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:15.416 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:15.416 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:15.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:15.759 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:15.759 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":526,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:15.780 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:15.780 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:16.061 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:16.066 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:16.080 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-18T14:15:16.045461+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003369"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70699999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.80300000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705772,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.52300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.77200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":1.008}]}]},{"osd":1,"up_from":0,"seq":38654705773,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705773,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:19.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:19.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:19.385 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:19.396 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-18T14:15:18.045879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.59399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.81499999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.876}]}]},{"osd":1,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:19.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:19.688 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:19.690 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:19.701 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":274,"stamp":"2024-09-18T14:15:18.045879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.59399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.81499999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.876}]}]},{"osd":1,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:19.702 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:19.702 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:19.702 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:19.702 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:19.702 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:20.038 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:20.038 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":530,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:20.052 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:20.052 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:20.338 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:20.352 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-18T14:15:20.046424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.59399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.81499999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.876}]}]},{"osd":1,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:23.354 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:23.653 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:23.655 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:23.666 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-18T14:15:22.046937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003147"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.59399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.81499999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.876}]}]},{"osd":1,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:23.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:23.954 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:23.957 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:23.968 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":276,"stamp":"2024-09-18T14:15:22.046937+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003147"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.59399999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705773,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.81499999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.58599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.876}]}]},{"osd":1,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705774,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:23.968 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:23.969 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:23.969 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:23.969 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:23.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:24.316 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:24.316 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":535,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:24.329 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:24.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:24.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:24.622 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:24.632 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2024-09-18T14:15:24.047456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.69199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.749},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.85799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.67900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.64000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.745},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.71499999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705775,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705775,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:27.634 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:27.934 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:27.937 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:27.947 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":278,"stamp":"2024-09-18T14:15:26.048049+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003207"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.69199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.749},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.85799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64900000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705774,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.67900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.64000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.745},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.71499999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705775,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705775,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:27.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:28.239 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:28.242 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:28.253 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-18T14:15:28.048584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003123"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.875}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.77000000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:28.254 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:28.254 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:28.254 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:28.254 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:28.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:28.602 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:28.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":539,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:28.616 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:28.617 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:28.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:28.902 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:28.913 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-18T14:15:28.048584+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003123"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.875}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.77000000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:31.915 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:32.209 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:32.212 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:32.223 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-18T14:15:32.049455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003031"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.875}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.77000000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:32.224 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:32.511 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:32.530 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-18T14:15:32.049455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003031"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.875}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.77000000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:32.530 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:32.530 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:32.531 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:32.531 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:32.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:32.869 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:32.870 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":543,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:32.883 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:32.883 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:33.168 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:33.171 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:33.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":281,"stamp":"2024-09-18T14:15:32.049455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003031"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.875}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72899999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705775,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.63800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.77000000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705776,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:36.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:36.487 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:36.491 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:36.501 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-18T14:15:36.050568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.80100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.76100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.65800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.755},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.81399999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705777,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705777,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:36.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:36.785 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:36.788 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:36.798 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-18T14:15:36.050568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.80100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.76100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.65800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.755},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.81399999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705777,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705777,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:36.799 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:36.799 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:36.799 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:36.800 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:36.800 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:37.152 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:37.152 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":548,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:37.166 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:37.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:37.460 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:37.463 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:37.474 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":283,"stamp":"2024-09-18T14:15:36.050568+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003112"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.80100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.76100000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705776,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.65800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.755},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.81399999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705777,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705777,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:40.476 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:40.775 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:40.777 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:40.788 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-18T14:15:40.051841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003256"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.66500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68799999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.76900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.71899999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705778,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705778,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:40.789 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:41.068 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:41.072 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:41.083 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-18T14:15:40.051841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003256"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.66500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68799999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.76900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.71899999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705778,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705778,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:41.084 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:41.084 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:41.084 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:41.084 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:41.084 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:41.437 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:41.437 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":552,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:41.451 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:41.451 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:41.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:41.739 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:41.749 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":285,"stamp":"2024-09-18T14:15:40.051841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003256"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.66500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68799999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705777,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.441}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.76900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.71899999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705778,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705778,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:44.751 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:45.050 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:45.053 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:45.063 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-18T14:15:44.053143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.752}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.79200000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.74099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.68300000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705779,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705779,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:45.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:45.351 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:45.357 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:45.364 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-18T14:15:44.053143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.752}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.79200000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.74099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.68300000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705779,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705779,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:45.365 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:45.365 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:45.365 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:45.365 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:45.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:45.713 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:45.713 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":556,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:45.732 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:45.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:46.020 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:46.023 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:46.033 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-18T14:15:44.053143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003688"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.752}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.79200000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705778,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.74099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.82199999999999995},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.68300000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705779,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705779,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:49.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:49.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:49.338 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:49.348 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-18T14:15:48.054261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003692"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.72399999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:49.350 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:49.638 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:49.641 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:49.652 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-09-18T14:15:48.054261+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003692"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.72399999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:49.653 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:49.653 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:49.653 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:49.653 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:49.654 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:50.000 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:50.000 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":560,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:50.019 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:50.019 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:50.300 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:50.303 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:50.314 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-09-18T14:15:50.054807+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003678"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.72399999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:53.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:53.621 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:53.624 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:53.635 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-18T14:15:52.055384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003543"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.72399999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:53.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:53.923 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:53.942 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":291,"stamp":"2024-09-18T14:15:52.055384+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003543"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.61099999999999999,"15min":0.61199999999999999},"min":{"1min":0.40100000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.86299999999999999,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.63300000000000001,"15min":0.63100000000000001},"min":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.88700000000000001,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.60399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.69099999999999995,"15min":0.69199999999999995},"min":{"1min":0.57799999999999996,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.97799999999999998,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68500000000000005,"15min":0.67800000000000005},"min":{"1min":0.44500000000000001,"5min":0.376,"15min":0.376},"max":{"1min":0.94999999999999996,"5min":1.034,"15min":1.034},"last":0.76000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:14:49 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.751,"15min":0.76000000000000001},"min":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.0169999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.71299999999999997,"15min":0.72499999999999998},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.85799999999999998,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68999999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705779,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.58899999999999997,"15min":0.60599999999999998},"min":{"1min":0.39100000000000001,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.80700000000000005,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.5},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.61399999999999999,"15min":0.63900000000000001},"min":{"1min":0.47099999999999997,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84399999999999997,"5min":0.88500000000000001,"15min":0.996},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66500000000000004},"min":{"1min":0.45100000000000001,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.88400000000000001,"5min":1.107,"15min":1.107},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65200000000000002,"15min":0.63700000000000001},"min":{"1min":0.47099999999999997,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.68300000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:14:50 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.76100000000000001,"15min":0.77900000000000003},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.84899999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78800000000000003,"15min":0.77300000000000002},"min":{"1min":0.57699999999999996,"5min":0.438,"15min":0.438},"max":{"1min":0.96299999999999997,"5min":1.1359999999999999,"15min":1.222},"last":0.72399999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705780,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:53.942 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:53.943 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:53.943 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:53.943 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:53.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:54.295 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:54.295 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":565,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:54.308 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:54.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:54.579 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:54.591 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2024-09-18T14:15:54.055864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003372"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.747}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.876}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.70699999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.83099999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705781,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705781,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:57.592 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:57.893 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:57.896 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:57.906 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2024-09-18T14:15:56.056411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003267"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.747}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.876}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.70699999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705780,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.83099999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705781,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705781,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:57.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:58.211 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-18T14:15:58.056864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.73699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:15:58.212 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:15:58.212 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:15:58.212 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:15:58.213 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:15:58.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:15:58.559 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:58.560 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":569,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:15:58.573 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:15:58.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:15:58.861 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:15:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:15:58.874 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":294,"stamp":"2024-09-18T14:15:58.056864+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.73699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:01.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:02.185 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:02.188 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:02.199 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-18T14:16:02.057812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003005"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.73699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:02.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:02.483 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:02.486 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:02.496 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-18T14:16:02.057812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003005"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.73699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:02.497 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:02.497 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:02.497 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:02.497 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:02.498 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:02.845 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:02.845 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":573,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:02.859 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:02.859 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:03.146 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:03.149 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:03.160 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2024-09-18T14:16:02.057812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003005"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.38900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.53400000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.60499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.73699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705782,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:06.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:06.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:06.460 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:06.476 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-18T14:16:06.058944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003079"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.749},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.79300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.66000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.40699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.79000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.74399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705783,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705783,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:06.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:06.758 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:06.761 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:06.776 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-18T14:16:06.058944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003079"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.749},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.79300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.66000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.40699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.79000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.74399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705783,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705783,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:06.776 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:06.777 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:06.777 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:06.777 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:06.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:07.124 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:07.124 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":578,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:07.137 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:07.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:07.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:07.429 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:07.440 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-09-18T14:16:06.058944+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003079"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.749},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.79300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.66000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.40699999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.79000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.74399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705783,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705783,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:10.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:10.742 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:10.745 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:10.755 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-18T14:16:10.060160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.623},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.747}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.46500000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61799999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705784,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705784,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:10.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:11.038 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:11.041 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:11.052 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-18T14:16:10.060160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.623},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.747}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.46500000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61799999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705784,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705784,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:11.054 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:11.054 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:11.054 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:11.054 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:11.055 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:11.402 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:11.403 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":582,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:11.416 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:11.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:11.702 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:11.705 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:11.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":300,"stamp":"2024-09-18T14:16:10.060160+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.623},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.71999999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.747}]}]},{"osd":2,"up_from":9,"seq":38654705783,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.46500000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61799999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.745}]}]},{"osd":1,"up_from":0,"seq":38654705784,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705784,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:14.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:15.015 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:15.026 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-18T14:16:14.061367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003554"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68500000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.497},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.72099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:15.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:15.316 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:15.319 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:15.330 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-18T14:16:14.061367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003554"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68500000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.497},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.72099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:15.331 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:15.331 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:15.331 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:15.331 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:15.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:15.672 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":586,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:15.691 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:15.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:15.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:15.977 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:15.994 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-09-18T14:16:14.061367+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003554"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.70699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.68500000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705784,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.497},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.72099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705785,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:18.996 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:19.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:19.301 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:19.312 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-18T14:16:18.062670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.748}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.98499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.79700000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:19.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:19.593 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:19.596 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:19.606 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":304,"stamp":"2024-09-18T14:16:18.062670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003726"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.748}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.98499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.79700000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:19.607 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:19.607 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:19.607 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:19.607 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:19.607 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:19.951 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:19.952 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":590,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:19.966 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:19.966 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:20.263 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:20.266 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:20.277 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":305,"stamp":"2024-09-18T14:16:20.063249+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.748}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.98499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.79700000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:23.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:23.576 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:23.589 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-18T14:16:22.063886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003725"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.748}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.98499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.79700000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:23.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:23.883 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:23.897 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":306,"stamp":"2024-09-18T14:16:22.063886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003725"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.748}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705785,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.501},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.98499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.79700000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705786,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:23.897 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:23.897 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:23.898 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:23.898 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:23.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:24.239 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:24.240 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":595,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:24.254 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:24.254 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:24.543 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:24.546 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:24.563 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":307,"stamp":"2024-09-18T14:16:24.064421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003708"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.87},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.56000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66200000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.629}]}]},{"osd":1,"up_from":0,"seq":38654705787,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705787,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:27.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:27.860 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:27.864 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:27.874 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-09-18T14:16:26.065090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003722"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.87},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.56000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66200000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.629}]}]},{"osd":1,"up_from":0,"seq":38654705787,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705787,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:27.876 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:28.164 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:28.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-18T14:16:28.065624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003555"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.87},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":1.099},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.69699999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:28.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:28.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:28.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:28.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:28.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:28.533 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:28.533 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":599,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:28.547 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:28.547 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:28.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:28.837 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:28.852 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":309,"stamp":"2024-09-18T14:16:28.065624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003555"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.87},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":1.099},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.69699999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:31.854 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:32.154 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:32.157 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:32.168 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-18T14:16:32.066729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.87},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":1.099},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.69699999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:32.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:32.452 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:32.466 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-18T14:16:32.066729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.87},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":1.099},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.69699999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:32.467 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:32.467 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:32.467 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:32.467 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:32.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:32.819 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:32.819 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":603,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:32.834 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:32.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:33.117 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:33.121 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:33.131 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":311,"stamp":"2024-09-18T14:16:32.066729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.70599999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.81000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.87},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705787,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":1.099},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.69699999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705788,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:36.133 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:36.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:36.434 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:36.445 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-18T14:16:36.067895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003474"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.73899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.751},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.88100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705789,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705789,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:36.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:36.724 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:36.727 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:36.737 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-18T14:16:36.067895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003474"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.73899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.751},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.88100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705789,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705789,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:36.738 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:36.738 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:36.738 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:36.739 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:36.739 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:37.086 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:37.086 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":607,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:37.099 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:37.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:37.383 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:37.386 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:37.396 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2024-09-18T14:16:36.067895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003474"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.73899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.64400000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705788,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.751},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.66200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.88100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705789,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705789,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:40.397 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:40.709 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-18T14:16:40.069112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003487"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.80300000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.755},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.878},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73599999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.79300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705790,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705790,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:40.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:40.991 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:41.004 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-18T14:16:40.069112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003487"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.80300000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.755},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.878},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73599999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.79300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705790,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705790,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:41.005 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:41.005 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:41.005 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:41.005 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:41.006 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:41.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:41.352 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":612,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:41.366 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:41.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:41.648 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:41.651 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:41.667 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":315,"stamp":"2024-09-18T14:16:40.069112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003487"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.80300000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.755},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.84199999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705789,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.878},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.621}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73599999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.79300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705790,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705790,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:44.668 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:44.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:44.970 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:44.981 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-18T14:16:44.070181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.89100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":1.1499999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.29},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":1.2170000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.78000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.89300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.753},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:44.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:45.287 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-18T14:16:44.070181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.89100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":1.1499999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.29},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":1.2170000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.78000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.89300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.753},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:45.288 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:45.288 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:45.288 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:45.288 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:45.289 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:45.631 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:45.631 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":616,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:45.645 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:45.646 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:45.936 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:45.939 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:45.949 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-09-18T14:16:44.070181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.89100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":1.0760000000000001},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":1.1499999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":1.29},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":1.2170000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705790,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.623},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.78000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.89300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.753},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705791,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:48.951 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:49.253 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:49.256 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:49.267 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-18T14:16:48.071426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003530"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.504},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.44}]}]},{"osd":1,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:49.268 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:49.554 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:49.558 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:49.568 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-09-18T14:16:48.071426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003530"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.504},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.44}]}]},{"osd":1,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:49.569 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:49.569 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:49.569 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:49.569 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:49.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:49.923 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:49.923 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":620,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:49.938 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:49.938 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:50.217 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:50.220 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:50.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":320,"stamp":"2024-09-18T14:16:50.072059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003590"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.504},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.44}]}]},{"osd":1,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:53.232 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:53.527 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:53.530 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:53.540 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-18T14:16:52.072460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003348"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.504},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.44}]}]},{"osd":1,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:53.541 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:53.831 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:53.912 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:53.912 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-09-18T14:16:52.072460+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003348"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.67600000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705791,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.504},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.34200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.44}]}]},{"osd":1,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705792,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:53.913 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:53.913 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:53.913 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:53.913 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:53.914 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:54.265 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:54.265 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":625,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:54.278 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:54.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:54.567 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:54.570 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:54.580 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":322,"stamp":"2024-09-18T14:16:54.072932+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003295"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.42499999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.748}]}]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.65900000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.76900000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705793,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705793,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:57.582 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:57.880 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:57.884 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:57.894 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2024-09-18T14:16:56.073421+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003239"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.61499999999999999,"15min":0.60899999999999999},"min":{"1min":0.39000000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83799999999999997,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.64200000000000002,"15min":0.63300000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.80100000000000005,"5min":0.96999999999999997,"15min":0.97399999999999998},"last":0.56399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.67900000000000005,"15min":0.68899999999999995},"min":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.95599999999999996,"5min":0.98099999999999998,"15min":1.0069999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.69299999999999995,"15min":0.68100000000000005},"min":{"1min":0.53500000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.876,"5min":1.034,"15min":1.034},"last":0.42499999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:15:50 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.751,"15min":0.75900000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.038,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.71499999999999997,"15min":0.72099999999999997},"min":{"1min":0.51300000000000001,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":0.82299999999999995,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.748}]}]},{"osd":2,"up_from":9,"seq":38654705792,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42999999999999999,"5min":0.38800000000000001,"15min":0.34499999999999997},"max":{"1min":0.78300000000000003,"5min":0.86399999999999999,"15min":0.93300000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.624,"5min":0.62,"15min":0.63700000000000001},"min":{"1min":0.441,"5min":0.39400000000000002,"15min":0.38200000000000001},"max":{"1min":0.81699999999999995,"5min":0.88500000000000001,"15min":0.996},"last":0.65900000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.52300000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.64200000000000002},"min":{"1min":0.56599999999999995,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.77900000000000003},"min":{"1min":0.58699999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.79300000000000004,"15min":0.77500000000000002},"min":{"1min":0.67800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.008,"5min":1.1359999999999999,"15min":1.222},"last":0.76900000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705793,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705793,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:57.895 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:58.182 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:58.185 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:58.195 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-18T14:16:58.073934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003155"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.5},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.81000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.69799999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:16:58.196 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:16:58.197 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:16:58.197 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:16:58.197 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:16:58.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:16:58.539 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:58.540 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":629,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:16:58.553 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:16:58.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:16:58.844 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:16:58.847 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:16:58.858 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":324,"stamp":"2024-09-18T14:16:58.073934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003155"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.5},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.81000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.69799999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:01.861 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:02.159 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:02.163 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:02.173 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-18T14:17:02.075010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002950"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.5},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.81000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.69799999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:02.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:02.460 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:02.463 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:02.473 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-18T14:17:02.075010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002950"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.5},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.81000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.69799999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:02.474 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:02.474 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:02.474 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:02.474 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:02.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:02.817 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:02.818 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":633,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:02.831 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:02.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:03.120 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:03.123 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:03.133 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-09-18T14:17:02.075010+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002950"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.5},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.81000000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705793,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.69799999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705794,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:06.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:06.438 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:06.452 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-18T14:17:06.076129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.623},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.60199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.70799999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705795,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705795,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:06.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:06.738 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:06.748 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-18T14:17:06.076129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.623},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.60199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.70799999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705795,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705795,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:06.749 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:06.749 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:06.750 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:06.750 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:06.750 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:07.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:07.099 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":637,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:07.124 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:07.124 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:07.413 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:07.416 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:07.426 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2024-09-18T14:17:06.076129+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003196"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.623},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.60199999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705794,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.70799999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705795,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705795,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:10.427 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:10.723 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:10.726 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:10.737 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-18T14:17:10.077235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003300"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.627},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.99399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.80100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.372},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.44900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.624},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.69899999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705796,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705796,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:10.738 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:11.026 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:11.029 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:11.039 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-18T14:17:10.077235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003300"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.627},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.99399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.80100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.372},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.44900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.624},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.69899999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705796,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705796,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:11.040 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:11.041 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:11.041 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:11.041 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:11.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:11.390 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:11.390 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":642,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:11.410 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:11.410 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:11.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:11.698 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:11.708 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":330,"stamp":"2024-09-18T14:17:10.077235+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003300"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.66800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.627},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.99399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.80100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705795,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.372},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.44900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.624},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.69899999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705796,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705796,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:14.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:15.008 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:15.011 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:15.022 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-18T14:17:14.078427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.82699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.749}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.71099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705797,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705797,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:15.023 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:15.309 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:15.312 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:15.323 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-18T14:17:14.078427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.82699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.749}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.71099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705797,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705797,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:15.324 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:15.324 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:15.324 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:15.324 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:15.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:15.672 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:15.672 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":646,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:15.686 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:15.686 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:15.970 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:15.974 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:15.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":332,"stamp":"2024-09-18T14:17:14.078427+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.82699999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705796,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.749}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.88200000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.71099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705797,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705797,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:18.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:19.284 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:19.287 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:19.297 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-18T14:17:18.079577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003447"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.377}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.88},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.74299999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:19.298 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:19.590 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:19.594 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:19.604 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":334,"stamp":"2024-09-18T14:17:18.079577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003447"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.377}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.88},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.74299999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:19.605 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:19.605 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:19.605 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:19.605 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:19.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:19.955 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:19.955 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":650,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:19.969 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:19.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:20.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:20.265 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:20.276 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":335,"stamp":"2024-09-18T14:17:20.080110+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.377}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.88},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.74299999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:23.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:23.574 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:23.576 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:23.586 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-18T14:17:22.080666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003431"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.377}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.88},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.74299999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:23.587 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:23.869 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:23.872 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:23.882 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":336,"stamp":"2024-09-18T14:17:22.080666+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003431"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.377}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.88},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.621}]}]},{"osd":2,"up_from":9,"seq":38654705797,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.74299999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705798,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:23.883 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:23.883 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:23.883 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:23.883 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:23.884 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:24.232 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:24.232 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":655,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:24.246 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:24.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:24.533 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:24.536 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:24.546 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":337,"stamp":"2024-09-18T14:17:24.081178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003406"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.77500000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.53600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.68799999999999994}]}]},{"osd":1,"up_from":0,"seq":38654705799,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705799,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:27.663 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:27.970 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:27.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-09-18T14:17:26.081766+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003339"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.77500000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.72599999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705798,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.53600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61199999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.86099999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.68799999999999994}]}]},{"osd":1,"up_from":0,"seq":38654705799,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705799,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:27.984 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:28.272 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:28.275 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:28.286 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-18T14:17:28.082239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003221"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.505},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.47399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.69599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.88900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:28.286 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:28.287 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:28.287 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:28.287 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:28.287 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:28.626 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:28.626 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":659,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:28.640 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:28.640 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:28.925 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:28.938 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-09-18T14:17:28.082239+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003221"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.505},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.47399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.69599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.88900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:31.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:32.244 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:32.249 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:32.264 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-18T14:17:32.083158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003048"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.505},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.47399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.69599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.88900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:32.265 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:32.556 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:32.569 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-18T14:17:32.083158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003048"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.505},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.47399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.69599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.88900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:32.570 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:32.570 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:32.570 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:32.570 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:32.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:32.915 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:32.915 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":663,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:32.934 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:32.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:33.227 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:33.231 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:33.241 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":341,"stamp":"2024-09-18T14:17:32.083158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003048"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.505},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.47399999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78200000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705799,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.69599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.88900000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705800,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:36.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:36.545 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:36.548 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:36.559 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-18T14:17:36.084523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003344"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.81599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.96699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.91800000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.70699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705801,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705801,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:36.560 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:36.842 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:36.845 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:36.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-18T14:17:36.084523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003344"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.81599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.96699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.91800000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.70699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705801,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705801,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:36.857 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:36.857 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:36.857 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:36.857 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:36.858 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:37.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:37.208 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":668,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:37.222 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:37.222 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:37.512 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:37.515 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:37.525 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":343,"stamp":"2024-09-18T14:17:36.084523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003344"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.81599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.96699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.91800000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705800,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.70699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705801,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705801,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:40.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:40.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:40.830 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:40.840 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-18T14:17:40.085588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003348"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.55100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":1.006}]}]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.65600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.76300000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705802,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705802,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:40.841 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:41.124 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:41.127 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:41.138 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-18T14:17:40.085588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003348"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.55100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":1.006}]}]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.65600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.76300000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705802,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705802,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:41.139 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:41.139 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:41.139 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:41.139 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:41.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:41.486 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:41.486 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":672,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:41.500 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:41.500 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:41.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:41.788 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:41.799 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-09-18T14:17:40.085588+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003348"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.55100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":1.006}]}]},{"osd":2,"up_from":9,"seq":38654705801,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.65600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.76300000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705802,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705802,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:44.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:45.094 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:45.097 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:45.107 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-18T14:17:44.086675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.74199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705803,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705803,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:45.108 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:45.395 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:45.399 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:45.409 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-18T14:17:44.086675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.74199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705803,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705803,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:45.410 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:45.410 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:45.410 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:45.410 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:45.411 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:45.750 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:45.750 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":676,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:45.764 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:45.764 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:46.054 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:46.057 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:46.069 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-09-18T14:17:44.086675+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003516"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.74199999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705802,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.64600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.75800000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705803,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705803,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:49.071 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:49.375 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:49.378 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:49.388 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-18T14:17:48.087803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003279"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.622},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.57099999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:49.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:49.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:49.681 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:49.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":349,"stamp":"2024-09-18T14:17:48.087803+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003279"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.622},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.57099999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:49.692 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:49.692 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:49.692 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:49.693 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:49.693 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:50.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:50.043 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":680,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:50.057 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:50.057 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:50.342 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:50.345 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:50.355 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2024-09-18T14:17:50.088385+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003356"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.622},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.57099999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:53.357 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:53.662 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:53.665 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:53.675 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-18T14:17:52.088925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003337"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.622},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.57099999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:53.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:53.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:53.969 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:53.979 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-18T14:17:52.088925+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003337"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.622},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.78000000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705803,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.57099999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.872}]}]},{"osd":1,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705804,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:53.980 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:53.980 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:53.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:53.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:53.981 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:54.324 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:54.324 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":685,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:54.337 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:54.337 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:54.619 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:54.622 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:54.633 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":352,"stamp":"2024-09-18T14:17:54.089443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003319"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705805,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705805,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:57.635 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:57.930 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:57.932 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:57.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":353,"stamp":"2024-09-18T14:17:56.090024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003349"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59999999999999998,"15min":0.60799999999999998},"min":{"1min":0.44900000000000001,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.83299999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.64400000000000002,"15min":0.63600000000000001},"min":{"1min":0.38900000000000001,"5min":0.379,"15min":0.379},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.58099999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.67900000000000005,"15min":0.68999999999999995},"min":{"1min":0.5,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.69899999999999995,"15min":0.68400000000000005},"min":{"1min":0.42499999999999999,"5min":0.376,"15min":0.376},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.75900000000000001},"min":{"1min":0.58999999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.75,"5min":0.71399999999999997,"15min":0.72199999999999998},"min":{"1min":0.53400000000000003,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.2170000000000001,"5min":3.0720000000000001,"15min":3.0720000000000001},"last":0.63200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705804,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.60799999999999998,"15min":0.60499999999999998},"min":{"1min":0.42599999999999999,"5min":0.39100000000000001,"15min":0.34499999999999997},"max":{"1min":1.099,"5min":1.099,"15min":1.099},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.61799999999999999,"15min":0.63300000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.996},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66400000000000003},"min":{"1min":0.48299999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.80000000000000004,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.66200000000000003,"15min":0.64400000000000002},"min":{"1min":0.46000000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.89300000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":0.754,"15min":0.77600000000000002},"min":{"1min":0.48899999999999999,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92000000000000004,"5min":1.085,"15min":1.085},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.751,"5min":0.78600000000000003,"15min":0.77400000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.438},"max":{"1min":0.88100000000000001,"5min":1.1359999999999999,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705805,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705805,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:57.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:58.226 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:58.231 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:58.245 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-18T14:17:58.090526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003255"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.871}]}]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.60099999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:17:58.246 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:17:58.246 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:17:58.246 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:17:58.246 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:17:58.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:17:58.592 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:58.592 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":689,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:17:58.606 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:17:58.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:17:58.897 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:17:58.900 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:17:58.911 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":354,"stamp":"2024-09-18T14:17:58.090526+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003255"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.871}]}]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.60099999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:01.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:02.206 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:02.209 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:02.220 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-18T14:18:02.091542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.871}]}]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.60099999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:02.221 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:02.511 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:02.514 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:02.525 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-18T14:18:02.091542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.871}]}]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.60099999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:02.526 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:02.526 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:02.526 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:02.526 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:02.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:02.872 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:02.872 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":693,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:02.886 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:02.887 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:03.177 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:03.180 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:03.192 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-09-18T14:18:02.091542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.871}]}]},{"osd":2,"up_from":9,"seq":38654705805,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.60099999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705806,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:06.193 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:06.488 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:06.491 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:06.501 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-18T14:18:06.092747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003303"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.871},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.78500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.755},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:06.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:06.793 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:06.798 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:06.810 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-18T14:18:06.092747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003303"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.871},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.78500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.755},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:06.811 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:06.811 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:06.811 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:06.811 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:06.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:07.153 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:07.153 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":698,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:07.167 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:07.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:07.446 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:07.449 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:07.459 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-09-18T14:18:06.092747+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003303"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.81899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.871},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.78500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705806,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67000000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.755},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.71799999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705807,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:10.460 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:10.748 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:10.751 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:10.762 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-18T14:18:10.093873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003346"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.90800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.94599999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.88600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:10.763 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:11.049 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:11.053 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:11.063 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-18T14:18:10.093873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003346"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.90800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.94599999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.88600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:11.064 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:11.064 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:11.064 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:11.064 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:11.064 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:11.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:11.407 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":702,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:11.421 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:11.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:11.712 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:11.725 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":360,"stamp":"2024-09-18T14:18:10.093873+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003346"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.90800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.94599999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705807,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.88600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705808,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:14.727 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:15.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:15.030 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:15.045 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-18T14:18:14.094945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.59099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.84299999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:15.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:15.333 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:15.336 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:15.347 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-18T14:18:14.094945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.59099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.84299999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:15.348 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:15.348 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:15.348 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:15.348 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:15.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:15.695 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:15.695 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":706,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:15.712 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:15.712 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:15.998 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:16.000 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:16.015 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-09-18T14:18:14.094945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003402"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67700000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.59099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705808,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66100000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.72599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.84299999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705809,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:19.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:19.314 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:19.317 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:19.328 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-18T14:18:18.096028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003281"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57099999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.75},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.61199999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:19.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:19.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:19.612 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:19.622 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":364,"stamp":"2024-09-18T14:18:18.096028+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003281"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57099999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.75},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.61199999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:19.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:19.623 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:19.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:19.624 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:19.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:19.957 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:19.957 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":710,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:19.971 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:19.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:20.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:20.263 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:20.273 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-09-18T14:18:20.096575+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003285"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57099999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.75},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.61199999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:23.275 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:23.573 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:23.576 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:23.587 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-18T14:18:22.097081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003208"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57099999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.75},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.61199999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:23.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:23.880 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:23.899 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":366,"stamp":"2024-09-18T14:18:22.097081+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003208"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705809,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67400000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57099999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.75},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.61199999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705810,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:23.900 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:23.900 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:23.900 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:23.900 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:23.900 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:24.245 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:24.245 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":715,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:24.260 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:24.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:24.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:24.552 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:24.562 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-09-18T14:18:24.097599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003216"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.67100000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:27.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:27.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:27.867 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:27.877 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":368,"stamp":"2024-09-18T14:18:26.098173+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003227"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.54700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.63400000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705810,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.67100000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705811,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:27.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:28.168 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:28.182 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-18T14:18:28.098667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003154"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.79000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61799999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.747},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":1.012}]}]},{"osd":1,"up_from":0,"seq":38654705812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:28.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:28.183 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:28.183 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:28.183 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:28.183 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:28.528 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:28.528 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":719,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:28.543 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:28.543 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:28.820 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:28.823 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:28.833 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":369,"stamp":"2024-09-18T14:18:28.098667+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003154"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.79000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61799999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.747},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":1.012}]}]},{"osd":1,"up_from":0,"seq":38654705812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:31.835 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:32.130 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:32.134 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:32.144 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-18T14:18:32.099837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.79000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61799999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.747},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":1.012}]}]},{"osd":1,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:32.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:32.437 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:32.440 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:32.450 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-18T14:18:32.099837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.79000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61799999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.747},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":1.012}]}]},{"osd":1,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:32.451 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:32.451 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:32.452 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:32.452 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:32.452 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:32.797 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:32.797 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":723,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:32.811 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:32.812 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:33.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:33.102 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:33.117 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-09-18T14:18:32.099837+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.52600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.79000000000000004}]}]},{"osd":2,"up_from":9,"seq":38654705811,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61799999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.747},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":1.012}]}]},{"osd":1,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705812,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:36.118 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:36.416 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:36.430 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-18T14:18:36.100999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003399"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.80100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.752},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.86399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705813,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:36.431 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:36.722 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:36.725 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:36.740 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-18T14:18:36.100999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003399"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.80100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.752},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.86399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705813,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:36.741 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:36.741 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:36.741 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:36.741 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:36.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:37.083 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:37.083 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":727,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:37.097 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:37.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:37.379 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:37.382 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:37.393 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":373,"stamp":"2024-09-18T14:18:36.100999+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003399"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73599999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705812,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.80100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.752},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.86399999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705813,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:40.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:40.697 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:40.700 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:40.716 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-18T14:18:40.102169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003502"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.439},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.51300000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.36899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67200000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:40.717 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:41.003 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:41.006 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:41.016 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-18T14:18:40.102169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003502"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.439},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.51300000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.36899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67200000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:41.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:41.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:41.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:41.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:41.017 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:41.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:41.364 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":732,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:41.377 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:41.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:41.661 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:41.665 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:41.678 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":375,"stamp":"2024-09-18T14:18:40.102169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003502"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.47999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.439},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.51300000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.36899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705813,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55600000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67200000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.86099999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705814,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:44.680 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:44.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:44.979 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:44.989 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-18T14:18:44.103253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.752}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.68100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:44.990 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:45.273 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:45.292 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-18T14:18:44.103253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.752}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.68100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:45.293 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:45.293 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:45.293 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:45.293 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:45.293 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:45.636 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:45.636 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":736,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:45.650 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:45.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:45.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:45.940 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:45.950 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":377,"stamp":"2024-09-18T14:18:44.103253+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003416"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.752}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.68100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705814,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705815,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:48.952 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:49.254 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:49.271 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-18T14:18:48.104516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003517"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.79200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.68200000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.79000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:49.272 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:49.563 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:49.566 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:49.576 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":379,"stamp":"2024-09-18T14:18:48.104516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003517"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.79200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.68200000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.79000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:49.577 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:49.577 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:49.577 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:49.577 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:49.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:49.922 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:49.922 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":740,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:49.936 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:49.936 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:50.246 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:50.260 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":380,"stamp":"2024-09-18T14:18:50.105090+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003533"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.79200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.68200000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.79000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:53.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:53.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:53.558 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:53.569 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-18T14:18:52.105624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003455"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.79200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.68200000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.79000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:53.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:53.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:53.859 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:53.869 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2024-09-18T14:18:52.105624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003455"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.79200000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.68200000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76200000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705815,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.57299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.79000000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705816,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:53.870 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:53.870 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:53.870 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:53.870 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:53.871 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:54.218 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":745,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:54.231 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:54.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:54.513 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:54.517 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:54.527 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":382,"stamp":"2024-09-18T14:18:54.106158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003448"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.89600000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:57.528 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:57.832 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:57.836 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:57.846 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":383,"stamp":"2024-09-18T14:18:56.106854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003600"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.60099999999999998,"15min":0.60599999999999998},"min":{"1min":0.47699999999999998,"5min":0.38200000000000001,"15min":0.33700000000000002},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.63600000000000001,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.84499999999999997,"5min":0.89100000000000001,"15min":0.97399999999999998},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.60699999999999998,"5min":0.41499999999999998,"15min":0.40200000000000002},"max":{"1min":0.88,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.70499999999999996,"15min":0.68600000000000005},"min":{"1min":0.51500000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:17:55 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.751,"15min":0.75800000000000001},"min":{"1min":0.623,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.98899999999999999,"5min":1.29,"15min":1.29},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.70299999999999996,"15min":0.72199999999999998},"min":{"1min":0.497,"5min":0.45100000000000001,"15min":0.41199999999999998},"max":{"1min":1.006,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.89600000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705816,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705817,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:57.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:58.136 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:58.139 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:58.150 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-18T14:18:58.107326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.83199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:18:58.151 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:18:58.152 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:18:58.152 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:18:58.152 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:18:58.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:18:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:58.503 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":749,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:18:58.524 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:18:58.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:18:58.818 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:18:58.821 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:18:58.832 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":384,"stamp":"2024-09-18T14:18:58.107326+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003453"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.83199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:01.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:02.126 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:02.129 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:02.139 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-18T14:19:02.108266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.83199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:02.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:02.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:02.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:02.442 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-18T14:19:02.108266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.83199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:02.443 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:02.443 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:02.443 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:02.443 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:02.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:02.788 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:02.788 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":753,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:02.802 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:02.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:03.089 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:03.092 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:03.103 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":386,"stamp":"2024-09-18T14:19:02.108266+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003176"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.83199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705817,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59999999999999998,"15min":0.60199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.82599999999999996,"5min":1.099,"15min":1.099},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.62,"15min":0.63300000000000001},"min":{"1min":0.41799999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.65700000000000003,"15min":0.66300000000000003},"min":{"1min":0.47699999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.94399999999999995,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.66200000000000003,"15min":0.64800000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.81899999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:17:56 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.752,"15min":0.77400000000000002},"min":{"1min":0.57599999999999996,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":0.92500000000000004,"5min":1.085,"15min":1.085},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.753,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.60899999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.88900000000000001,"5min":1.008,"15min":1.222},"last":0.90500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705818,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:06.105 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:06.402 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:06.416 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-18T14:19:06.109358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.88500000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.69999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:06.417 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:06.704 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:06.707 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:06.718 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-18T14:19:06.109358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.88500000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.69999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:06.719 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:06.719 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:06.719 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:06.719 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:06.719 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:07.060 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:07.061 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":757,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:07.075 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:07.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:07.361 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:07.364 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:07.375 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":388,"stamp":"2024-09-18T14:19:06.109358+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003199"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56799999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.88500000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705818,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.69999999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705819,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:10.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:10.674 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:10.677 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:10.688 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-18T14:19:10.110492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.68100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.626}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.86499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:10.690 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:10.972 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:10.976 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:10.986 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-18T14:19:10.110492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.68100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.626}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.86499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:10.988 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:10.988 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:10.988 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:10.988 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:10.989 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:11.330 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:11.330 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":762,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:11.343 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:11.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:11.628 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:11.632 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:11.643 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-09-18T14:19:10.110492+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003166"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.68100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705819,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.626}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.86499999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705820,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:14.645 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:14.938 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:14.952 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-18T14:19:14.111587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003320"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77300000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.76800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68700000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:14.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:15.242 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:15.245 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:15.258 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-18T14:19:14.111587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003320"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77300000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.76800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68700000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:15.259 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:15.259 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:15.259 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:15.259 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:15.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:15.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:15.601 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":766,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:15.615 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:15.615 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:15.901 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:15.904 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:15.914 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-09-18T14:19:14.111587+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003320"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65800000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.77300000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81100000000000005}]}]},{"osd":2,"up_from":9,"seq":38654705820,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.76800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.68700000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705821,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:18.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:19:19.234+0000 7fbdcb7fe700 0 --2- 172.21.15.16:0/1595879750 >> [v2:172.21.15.16:3302/0,v1:172.21.15.16:6791/0] conn(0x7fbdcc06bdd0 0x7fbdcc126b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:19:19.424 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:19.427 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:19.442 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-18T14:19:18.112758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.59999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.628},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.72299999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:19.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:19.732 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:19.734 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:19.745 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":394,"stamp":"2024-09-18T14:19:18.112758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003400"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.59999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.628},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.72299999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:19.746 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:19.746 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:19.746 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:19.746 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:19.747 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:20.090 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:20.090 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":770,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:20.103 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:20.104 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:20.389 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:20.392 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:20.403 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":395,"stamp":"2024-09-18T14:19:20.113270+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003353"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.59999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.628},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.72299999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:23.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:23.703 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:23.716 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-18T14:19:22.113676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003183"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.59999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.628},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.72299999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:23.717 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:23.994 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:23.997 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:24.007 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":396,"stamp":"2024-09-18T14:19:22.113676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003183"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65000000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.75700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.59999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705821,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.628},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.72299999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705822,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:24.008 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:24.008 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:24.008 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:24.008 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:24.009 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:24.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:24.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":775,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:24.367 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:24.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:24.658 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:24.661 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:24.672 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":397,"stamp":"2024-09-18T14:19:24.114165+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.83999999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.629}]}]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.49399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.86699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:27.673 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:27.969 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:27.982 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":398,"stamp":"2024-09-18T14:19:26.114714+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003126"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.83999999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.629}]}]},{"osd":2,"up_from":9,"seq":38654705822,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.49399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.86699999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705823,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:27.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:28.281 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:28.283 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:28.294 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-18T14:19:28.115220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003046"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.625},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.65700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.749},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:28.295 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:28.295 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:28.295 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:28.295 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:28.296 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:28.641 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:28.642 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":779,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":7,"modified":"2024-09-18T14:14:24.031107+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:28.656 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:28.656 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:28.947 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:28.950 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:28.961 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-09-18T14:19:28.115220+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003046"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.625},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.65700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.749},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:31.963 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:32.266 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:32.269 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:32.279 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-18T14:19:32.116188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.625},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.65700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.749},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:32.280 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:32.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:32.560 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:32.570 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-18T14:19:32.116188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.625},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.65700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.749},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:32.571 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:32.571 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:32.571 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:32.571 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:32.571 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:32.916 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:32.916 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":783,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:32.937 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:32.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:33.219 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:33.221 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:33.232 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":401,"stamp":"2024-09-18T14:19:32.116188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002917"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58899999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.625},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.78500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.65700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705823,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.68999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.749},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705824,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:36.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:36.537 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:36.540 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:36.550 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-18T14:19:36.117274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003109"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.83299999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:36.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:36.832 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:36.846 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-18T14:19:36.117274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003109"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.83299999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:36.846 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:36.847 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:36.847 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:36.847 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:36.847 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:37.189 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:37.189 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":788,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:37.209 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:37.209 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:37.491 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:37.494 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:37.504 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":403,"stamp":"2024-09-18T14:19:36.117274+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003109"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.61799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705824,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.83299999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705825,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:40.506 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:40.801 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:40.804 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:40.815 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-18T14:19:40.118366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003146"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.86699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.748},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:40.816 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:41.101 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:41.104 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:41.115 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-18T14:19:40.118366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003146"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.86699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.748},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:41.116 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:41.116 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:41.116 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:41.116 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:41.117 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:41.460 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:41.461 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":792,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:41.475 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:41.475 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:41.761 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:41.763 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:41.780 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":405,"stamp":"2024-09-18T14:19:40.118366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003146"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72499999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.86699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705825,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.622}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.748},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.82699999999999996}]}]},{"osd":1,"up_from":0,"seq":38654705827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705826,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:44.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:45.067 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:45.070 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:45.081 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-18T14:19:44.119450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.752}]}]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.44},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.75},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.63700000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:45.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:45.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:45.374 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:45.385 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-18T14:19:44.119450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.752}]}]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.44},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.75},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.63700000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:45.386 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:45.386 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:45.386 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:45.386 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:45.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:45.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:45.731 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":796,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:45.745 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:45.745 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:46.038 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:46.042 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:46.053 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":407,"stamp":"2024-09-18T14:19:44.119450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003261"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60599999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.752}]}]},{"osd":2,"up_from":9,"seq":38654705826,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.44},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.75},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.63700000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705827,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:49.054 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:49.351 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:49.353 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:49.364 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-18T14:19:48.120553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003278"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.495},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.625},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.78500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:49.365 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:49.647 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:49.650 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:49.661 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":409,"stamp":"2024-09-18T14:19:48.120553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003278"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.495},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.625},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.78500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:49.662 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:49.662 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:49.662 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:49.662 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:49.662 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:50.010 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:50.011 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":800,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:50.028 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:50.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:50.313 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:50.316 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:50.327 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":410,"stamp":"2024-09-18T14:19:50.121108+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.495},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.625},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.78500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:53.329 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:53.622 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:53.625 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:53.635 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-18T14:19:52.121547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003181"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.495},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.625},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.78500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:53.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:53.918 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:53.921 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:53.932 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":411,"stamp":"2024-09-18T14:19:52.121547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003181"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.44},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72099999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.495},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.66300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705827,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.625},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.78500000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705828,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:53.933 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:53.934 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:53.934 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:53.934 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:53.934 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:54.283 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:54.284 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":805,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:54.301 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:54.302 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:54.588 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:54.592 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:54.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":412,"stamp":"2024-09-18T14:19:54.122146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003243"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.73099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.69199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.52400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.504},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.65100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:57.604 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:57.896 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:57.899 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:57.910 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2024-09-18T14:19:56.122682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003232"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.73099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.69199999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705828,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.52400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.60499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.504},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.65100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705829,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:57.912 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:58.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:58.200 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:58.217 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-18T14:19:58.123282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003275"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.83799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705831,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:19:58.218 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:19:58.218 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:19:58.219 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:19:58.219 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:19:58.219 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:19:58.575 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:58.575 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":809,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:19:58.589 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:19:58.589 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:19:58.871 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:19:58.874 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:19:58.884 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":414,"stamp":"2024-09-18T14:19:58.123282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003275"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.83799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705831,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:01.886 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:02.183 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:02.186 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:02.196 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-18T14:20:02.124449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003341"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.83799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:02.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:02.488 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:02.491 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:02.507 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-18T14:20:02.124449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003341"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.83799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:02.508 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:02.508 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:02.508 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:02.508 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:02.508 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:02.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:02.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":813,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:02.874 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:02.875 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:03.164 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:03.167 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:03.178 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":416,"stamp":"2024-09-18T14:20:02.124449+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003341"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60399999999999998,"15min":0.60499999999999998},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":0.95499999999999996},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63300000000000001,"15min":0.63600000000000001},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.70899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.67200000000000004,"15min":0.68500000000000005},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93700000000000006,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.71199999999999997,"15min":0.68799999999999994},"min":{"1min":0.51300000000000001,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.84299999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:18:57 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.754,"15min":0.75600000000000001},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88900000000000001,"5min":1.29,"15min":1.29},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.71299999999999997,"15min":0.71999999999999997},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.94599999999999995,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60699999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705829,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.60499999999999998,"15min":0.60199999999999998},"min":{"1min":0.46899999999999997,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":1.099,"15min":1.099},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.63600000000000001,"15min":0.63300000000000001},"min":{"1min":0.48199999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.66400000000000003,"15min":0.66500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.33200000000000002},"max":{"1min":0.85599999999999998,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.66500000000000004,"15min":0.64900000000000002},"min":{"1min":0.52100000000000002,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.77100000000000002,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:18:58 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.76000000000000001,"15min":0.77200000000000002},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.93100000000000005,"5min":1.085,"15min":1.085},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.60099999999999998,"5min":0.44,"15min":0.438},"max":{"1min":1.012,"5min":1.012,"15min":1.222},"last":0.83799999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705830,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:06.179 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:06.482 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:06.485 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:06.496 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-18T14:20:06.125697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.95199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.54000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705831,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:06.497 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:06.782 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:06.785 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:06.795 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-18T14:20:06.125697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.95199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.54000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705831,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:06.796 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:06.796 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:06.796 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:06.797 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:06.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:07.149 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:07.149 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":818,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:07.167 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:07.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:07.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:07.458 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:07.470 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":418,"stamp":"2024-09-18T14:20:06.125697+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69999999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.82399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.95199999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705830,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.71599999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.54000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705831,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:10.471 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:10.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:10.775 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:10.786 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-18T14:20:10.127016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003733"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.38}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:10.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:11.076 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:11.079 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:11.090 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-18T14:20:10.127016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003733"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.38}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:11.090 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:11.091 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:11.091 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:11.091 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:11.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:11.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":822,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:11.441 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:11.441 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:11.719 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:11.733 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-09-18T14:20:10.127016+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003733"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.38}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56899999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705831,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.69599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705832,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:14.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:15.038 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:15.041 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:15.052 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-18T14:20:14.128053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64500000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.80900000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.56499999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.72099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:15.053 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:15.338 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:15.341 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:15.351 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-18T14:20:14.128053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64500000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.80900000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.56499999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.72099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:15.352 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:15.352 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:15.352 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:15.352 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:15.352 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:15.689 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:15.689 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":826,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:15.703 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:15.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:15.990 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:15.993 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:16.003 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-18T14:20:14.128053+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64500000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.80900000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705832,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.56499999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.72099999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705833,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:19.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:19.303 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:19.305 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:19.316 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-18T14:20:18.129142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003444"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.748},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.89100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:19.317 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:19.602 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:19.606 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:19.616 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":424,"stamp":"2024-09-18T14:20:18.129142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003444"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.748},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.89100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:19.617 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:19.617 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:19.617 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:19.617 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:19.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:19.962 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:19.963 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":830,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:19.983 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:19.983 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:20.274 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:20.278 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:20.289 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":425,"stamp":"2024-09-18T14:20:20.129658+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003320"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.748},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.89100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:23.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:23.587 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:23.590 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:23.600 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-18T14:20:22.130126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003110"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.748},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.89100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:23.601 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:23.887 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:23.900 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-09-18T14:20:22.130126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003110"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.748},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705833,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69299999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.89100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705834,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:23.901 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:23.901 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:23.902 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:23.902 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:23.902 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:24.246 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:24.246 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":835,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:24.259 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:24.260 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:24.551 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:24.554 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:24.564 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-09-18T14:20:24.130644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.748},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.77800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.90300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:27.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:27.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:27.867 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:27.878 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":428,"stamp":"2024-09-18T14:20:26.131230+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003177"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.46999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.748},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.76600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705834,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.77800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.90300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705835,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:27.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:28.161 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:28.164 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:28.180 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-18T14:20:28.131743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.77400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.751}]}]},{"osd":1,"up_from":0,"seq":38654705837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:28.181 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:28.181 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:28.181 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:28.181 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:28.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:28.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:28.525 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":839,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:28.538 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:28.538 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:28.816 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:28.819 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:28.829 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":429,"stamp":"2024-09-18T14:20:28.131743+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003125"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.77400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.751}]}]},{"osd":1,"up_from":0,"seq":38654705837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:31.830 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:32.126 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:32.129 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:32.139 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":430,"stamp":"2024-09-18T14:20:30.132287+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003145"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.77400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.751}]}]},{"osd":1,"up_from":0,"seq":38654705837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:32.141 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:32.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:32.429 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:32.445 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-18T14:20:32.132791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.77400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.751}]}]},{"osd":1,"up_from":0,"seq":38654705838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:32.445 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:32.446 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:32.446 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:32.446 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:32.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:32.796 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:32.796 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":843,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:32.811 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:32.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:33.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:33.103 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:33.113 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":431,"stamp":"2024-09-18T14:20:32.132791+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003133"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.77400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.60999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56200000000000006}]}]},{"osd":2,"up_from":9,"seq":38654705835,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.63800000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55900000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.751}]}]},{"osd":1,"up_from":0,"seq":38654705838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705836,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:36.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:36.417 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:36.419 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:36.435 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-18T14:20:36.133890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003246"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.64500000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.45800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.371},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.40600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.505},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.626},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.68200000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:36.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:36.732 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:36.735 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:36.745 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-18T14:20:36.133890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003246"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.64500000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.45800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.371},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.40600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.505},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.626},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.68200000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:36.746 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:36.746 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:36.746 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:36.746 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:36.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:37.093 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:37.093 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":847,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:37.106 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:37.106 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:37.396 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:37.399 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:37.410 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2024-09-18T14:20:36.133890+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003246"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.64500000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.45800000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705836,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.371},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.40600000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.505},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.55500000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.626},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.68200000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705837,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:40.413 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:40.712 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:40.715 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:40.725 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-18T14:20:40.135009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003266"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:40.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:41.014 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:41.017 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:41.027 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-18T14:20:40.135009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003266"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:41.028 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:41.028 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:41.028 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:41.028 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:41.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:41.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:41.373 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":852,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:41.386 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:41.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:41.675 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:41.678 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:41.689 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":435,"stamp":"2024-09-18T14:20:40.135009+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003266"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705837,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705838,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:44.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:44.991 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:44.995 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:45.004 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-18T14:20:44.136112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003320"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:45.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:45.294 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:45.297 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:45.307 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-18T14:20:44.136112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003320"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:45.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:45.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:45.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:45.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:45.310 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:45.664 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:45.664 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":856,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:45.678 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:45.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:45.964 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:45.967 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:45.978 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":437,"stamp":"2024-09-18T14:20:44.136112+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003320"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.74399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705838,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61599999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.65800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.751},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.84999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705839,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:48.980 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:49.281 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:49.283 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:49.293 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-09-18T14:20:48.137243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003352"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.54800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.626},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:49.294 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:49.582 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:49.595 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":439,"stamp":"2024-09-18T14:20:48.137243+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003352"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.54800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.626},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:49.596 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:49.597 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:49.597 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:49.597 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:49.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:49.947 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:49.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":860,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:49.961 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:49.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:50.258 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:50.272 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":440,"stamp":"2024-09-18T14:20:50.137797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003360"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.54800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.626},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:53.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:53.574 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:53.576 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:53.589 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-09-18T14:20:52.138224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.54800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.626},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:53.590 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:53.878 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:53.883 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:53.897 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":441,"stamp":"2024-09-18T14:20:52.138224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003214"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.54800000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.626},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60099999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705839,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705840,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:53.898 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:53.898 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:53.898 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:53.898 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:53.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:54.246 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:54.246 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":865,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:54.265 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:54.265 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:54.550 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:54.553 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:54.563 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":442,"stamp":"2024-09-18T14:20:54.138677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003117"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.77200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:57.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:57.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:57.866 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:57.877 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-18T14:20:56.139144+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003032"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.77200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.67100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705840,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705841,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:57.878 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:58.166 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:58.172 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:58.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-18T14:20:58.139652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002936"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.69399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:20:58.181 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:20:58.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:20:58.182 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:20:58.182 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:20:58.182 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:20:58.529 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:58.530 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":869,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:20:58.544 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:20:58.544 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:20:58.834 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:20:58.838 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:20:58.848 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":444,"stamp":"2024-09-18T14:20:58.139652+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002936"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.69399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:01.850 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:02.142 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:02.145 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:02.155 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":445,"stamp":"2024-09-18T14:21:00.140227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002983"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.69399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:02.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:02.445 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:02.449 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:02.459 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-18T14:21:02.140682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.69399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:02.460 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:02.460 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:02.460 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:02.461 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:02.461 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:02.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:02.806 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":873,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:02.824 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:02.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:03.114 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:03.117 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:03.128 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":446,"stamp":"2024-09-18T14:21:02.140682+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.63}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72599999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.69399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705841,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.44,"5min":0.372,"15min":0.34499999999999997},"max":{"1min":0.74099999999999999,"5min":1.099,"15min":1.099},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.63,"15min":0.63},"min":{"1min":0.43099999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.76800000000000002,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85899999999999999,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.66400000000000003,"15min":0.65000000000000002},"min":{"1min":0.48799999999999999,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73499999999999999,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:19:58 2024","interfaces":[{"interface":"back","average":{"1min":0.73799999999999999,"5min":0.76100000000000001,"15min":0.76800000000000002},"min":{"1min":0.504,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82099999999999995,"5min":1.085,"15min":1.085},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77000000000000002,"15min":0.77500000000000002},"min":{"1min":0.49199999999999999,"5min":0.44,"15min":0.438},"max":{"1min":0.93400000000000005,"5min":1.012,"15min":1.222},"last":0.83599999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705842,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:06.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:06.434 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:06.437 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:06.448 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-18T14:21:06.141674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002996"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.88600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.70999999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:06.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:06.745 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:06.748 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:06.757 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-18T14:21:06.141674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002996"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.88600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.70999999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:06.758 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:06.758 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:06.759 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:06.759 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:06.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:07.105 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:07.105 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":877,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:07.119 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:07.119 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:07.405 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:07.408 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:07.419 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":448,"stamp":"2024-09-18T14:21:06.141674+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002996"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59199999999999997,"15min":0.60399999999999998},"min":{"1min":0.44,"5min":0.38700000000000001,"15min":0.33700000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64400000000000002,"15min":0.63900000000000001},"min":{"1min":0.56799999999999995,"5min":0.377,"15min":0.377},"max":{"1min":0.78100000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.88600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.66200000000000003,"15min":0.68300000000000005},"min":{"1min":0.39400000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.90700000000000003,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.745,"5min":0.71999999999999997,"15min":0.69199999999999995},"min":{"1min":0.60599999999999998,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.86699999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:20:01 2024","interfaces":[{"interface":"back","average":{"1min":0.76900000000000002,"5min":0.755,"15min":0.75700000000000001},"min":{"1min":0.495,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.94099999999999995,"5min":1.29,"15min":1.29},"last":0.85399999999999998},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.59999999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.95199999999999996,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.81499999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705842,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60899999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.70999999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77300000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705843,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:10.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:10.721 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:10.724 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:10.734 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-18T14:21:10.142741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003089"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.503},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.40300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.626}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77200000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:10.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:11.022 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:11.026 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:11.036 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-18T14:21:10.142741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003089"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.503},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.40300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.626}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77200000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:11.037 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:11.037 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:11.037 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:11.037 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:11.038 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:11.385 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:11.385 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":882,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:11.399 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:11.400 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:11.689 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:11.692 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:11.702 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":450,"stamp":"2024-09-18T14:21:10.142741+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003089"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.503},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.40300000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.83699999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705843,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.626}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77200000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705844,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:14.703 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:14.996 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:14.996 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:15.007 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-18T14:21:14.143804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003121"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.61499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:15.008 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:15.297 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:15.299 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:15.314 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-18T14:21:14.143804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003121"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.61499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:15.315 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:15.315 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:15.315 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:15.315 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:15.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:15.659 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:15.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":886,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:15.673 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:15.673 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:15.957 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:15.960 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:15.971 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":452,"stamp":"2024-09-18T14:21:14.143804+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003121"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69699999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.57899999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.753},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.61499999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705844,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.59999999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.64100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705845,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:18.973 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:19.270 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:19.273 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:19.284 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-09-18T14:21:18.144947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003273"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77900000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:19.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:19.568 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:19.571 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:19.582 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":454,"stamp":"2024-09-18T14:21:18.144947+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003273"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77900000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:19.583 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:19.583 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:19.583 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:19.583 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:19.583 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:19.941 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:19.941 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":890,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:19.959 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:19.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:20.248 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:20.252 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:20.262 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":455,"stamp":"2024-09-18T14:21:20.145513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003344"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77900000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:23.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:23.565 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:23.578 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-09-18T14:21:22.146046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003304"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77900000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:23.580 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:23.870 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:23.873 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:23.889 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":456,"stamp":"2024-09-18T14:21:22.146046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003304"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.68000000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.88900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73399999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705845,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.67500000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.77900000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705846,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:23.890 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:23.890 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:23.890 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:23.890 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:23.891 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:24.234 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:24.234 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":895,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:24.248 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:24.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:24.538 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:24.541 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:24.552 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":457,"stamp":"2024-09-18T14:21:24.146550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003275"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65400000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.84599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.58199999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:27.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:27.854 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:27.857 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:27.868 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-18T14:21:26.147118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003313"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65400000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.84599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.58199999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:27.869 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:28.149 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:28.152 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:28.163 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":458,"stamp":"2024-09-18T14:21:26.147118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003313"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.65400000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.84599999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705846,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.61299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.58199999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76600000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705847,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:28.164 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:28.164 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:28.164 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:28.164 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:28.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:28.513 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:28.514 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":899,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:28.527 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:28.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:28.811 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:28.814 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:28.824 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":459,"stamp":"2024-09-18T14:21:28.147639+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003247"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.879},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81899999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:31.826 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:32.122 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:32.125 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:32.137 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":460,"stamp":"2024-09-18T14:21:30.148217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003269"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.879},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81899999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:32.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:32.425 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:32.428 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:32.439 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-18T14:21:32.148750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003236"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.879},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81899999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:32.440 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:32.440 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:32.440 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:32.440 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:32.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:32.791 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:32.791 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":903,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:32.810 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:32.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:33.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:33.102 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:33.112 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":461,"stamp":"2024-09-18T14:21:32.148750+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003236"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.79400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.73799999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705847,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.879},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.60699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81899999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705848,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:36.114 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:36.415 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:36.418 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:36.429 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-18T14:21:36.149784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003233"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.73199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.496},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.42499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.72799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:36.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:36.720 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:36.723 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:36.733 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-18T14:21:36.149784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003233"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.73199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.496},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.42499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.72799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:36.734 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:36.734 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:36.734 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:36.734 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:36.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:37.071 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:37.071 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":907,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:37.092 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:37.092 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:37.382 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:37.385 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:37.394 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":463,"stamp":"2024-09-18T14:21:36.149784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003233"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.627}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.73199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.84299999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705848,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.496},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.40500000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.42499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.72799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705849,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:40.396 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:40.693 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:40.698 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:40.708 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-18T14:21:40.151071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003432"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.71999999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.69599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:40.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:40.990 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:41.009 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-18T14:21:40.151071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003432"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.71999999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.69599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:41.010 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:41.010 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:41.010 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:41.010 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:41.011 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:41.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:41.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":912,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:41.367 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:41.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:41.654 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:41.657 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:41.668 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2024-09-18T14:21:40.151071+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003432"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56999999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.71999999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705849,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.69599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.74299999999999999}]}]},{"osd":1,"up_from":0,"seq":38654705851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705850,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:44.670 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:44.971 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:44.974 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:44.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-18T14:21:44.152413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003663"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76900000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:44.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:45.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:45.273 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:45.283 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-18T14:21:44.152413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003663"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76900000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:45.284 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:45.284 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:45.284 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:45.284 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:45.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:45.634 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:45.634 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":916,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:45.648 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:45.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:45.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:45.936 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:45.946 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":467,"stamp":"2024-09-18T14:21:44.152413+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003663"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.64800000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60799999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705850,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.45600000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.76900000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705851,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:48.947 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:49.252 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:49.255 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:49.265 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2024-09-18T14:21:48.153590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.80500000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.85299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:49.266 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:49.549 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:49.562 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":469,"stamp":"2024-09-18T14:21:48.153590+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.80500000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.85299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:49.563 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:49.563 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:49.563 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:49.563 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:49.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:49.902 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:49.902 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":920,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:49.916 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:49.916 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:50.198 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:50.201 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:50.211 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":470,"stamp":"2024-09-18T14:21:50.154167+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003805"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.80500000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.85299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:53.213 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:53.508 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:53.511 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:53.526 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-18T14:21:52.154810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003739"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.80500000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.85299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:53.527 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:53.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:53.826 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":471,"stamp":"2024-09-18T14:21:52.154810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003739"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.80500000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.85299999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705851,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.60999999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.56599999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705852,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:53.827 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:53.827 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:53.827 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:53.828 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:53.828 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:54.177 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:54.177 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":925,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:54.191 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:54.191 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:54.473 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:54.477 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:54.487 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":472,"stamp":"2024-09-18T14:21:54.155380+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003632"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.80900000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:57.489 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:57.784 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:57.787 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:57.797 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-09-18T14:21:56.155841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003427"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.80900000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:57.798 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:58.086 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:58.090 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:58.100 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":473,"stamp":"2024-09-18T14:21:56.155841+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003427"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.71999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.85999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.77200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705852,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.66000000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.80900000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705853,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:21:58.101 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:21:58.101 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:21:58.101 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:21:58.101 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:21:58.102 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:21:58.442 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:58.442 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":929,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:21:58.455 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:21:58.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:21:58.742 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:21:58.745 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:21:58.755 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":474,"stamp":"2024-09-18T14:21:58.156395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003371"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60399999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.747},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.84399999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:01.757 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:02.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:02.072 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2024-09-18T14:22:00.156965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003374"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60399999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.747},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.84399999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:02.073 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:02.359 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:02.362 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:02.373 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-18T14:22:02.157439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003271"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60399999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.747},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.84399999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:02.374 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:02.374 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:02.374 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:02.374 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:02.374 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:02.720 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:02.721 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":933,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:02.734 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:02.735 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:03.021 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:03.024 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:03.034 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":476,"stamp":"2024-09-18T14:22:02.157439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003271"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.84399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.65000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.60399999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705853,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58999999999999997,"15min":0.59399999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":1.099,"15min":1.099},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.629,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.40600000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.82899999999999996,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.55700000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.87,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.55500000000000005,"5min":0.44900000000000001,"15min":0.41599999999999998},"max":{"1min":0.73799999999999999,"5min":0.89300000000000002,"15min":0.96699999999999997},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:21:00 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.76000000000000001,"15min":0.76700000000000002},"min":{"1min":0.626,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.92400000000000004,"5min":0.93100000000000005,"15min":1.085},"last":0.747},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.68200000000000005,"5min":0.44,"15min":0.438},"max":{"1min":0.90300000000000002,"5min":1.012,"15min":1.222},"last":0.84399999999999997}]}]},{"osd":1,"up_from":0,"seq":38654705856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705854,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:06.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:06.341 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:06.355 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-18T14:22:06.158407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.76600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.53500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.499},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.438}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.69099999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.76300000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:06.356 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:06.645 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:06.648 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:06.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-18T14:22:06.158407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.76600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.53500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.499},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.438}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.69099999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.76300000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:06.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:06.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:06.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:06.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:06.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:07.001 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:07.001 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":937,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:07.021 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:07.021 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:07.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:07.308 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:07.318 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":478,"stamp":"2024-09-18T14:22:06.158407+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003026"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.76600000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.69199999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.53500000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.499},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.438}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.69099999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.76300000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:10.320 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:10.618 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:10.620 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:10.635 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-18T14:22:10.159698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.49299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.755},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.89100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:10.636 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:10.922 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:10.925 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:10.936 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-18T14:22:10.159698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.49299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.755},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.89100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:10.936 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:10.937 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:10.937 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:10.937 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:10.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:11.285 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:11.285 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":942,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:11.299 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:11.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:11.578 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:11.592 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":480,"stamp":"2024-09-18T14:22:10.159698+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.59499999999999997,"15min":0.60199999999999998},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.79500000000000004,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.63300000000000001,"15min":0.63800000000000001},"min":{"1min":0.38,"5min":0.377,"15min":0.377},"max":{"1min":0.88600000000000001,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.49299999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.66200000000000003,"15min":0.67900000000000005},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81000000000000005,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.71399999999999997,"15min":0.69299999999999995},"min":{"1min":0.59599999999999997,"5min":0.42499999999999999,"15min":0.376},"max":{"1min":0.83799999999999997,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:21:06 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.752,"15min":0.755},"min":{"1min":0.47699999999999998,"5min":0.42799999999999999,"15min":0.35199999999999998},"max":{"1min":0.86399999999999999,"5min":1.29,"15min":1.29},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.72099999999999997,"15min":0.71599999999999997},"min":{"1min":0.45800000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.84199999999999997,"5min":1.2170000000000001,"15min":3.0720000000000001},"last":0.56399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69599999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.755},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.89100000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705856,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:14.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:14.892 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:14.902 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-09-18T14:22:14.160852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003413"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.745}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.64900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.82499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.499},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.42399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.72699999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:14.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:15.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:15.195 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:15.206 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-09-18T14:22:14.160852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003413"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.745}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.64900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.82499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.499},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.42399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.72699999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:15.207 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:15.207 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:15.207 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:15.207 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:15.207 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:15.556 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:15.556 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":946,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:15.569 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:15.570 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:15.871 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":482,"stamp":"2024-09-18T14:22:14.160852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003413"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.745}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.64900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.82499999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.499},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.42399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.72699999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:18.873 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:19.169 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:19.172 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:19.188 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-09-18T14:22:18.162041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003633"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.78700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.752},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:19.189 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:19.476 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:19.479 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:19.489 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-09-18T14:22:18.162041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003633"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.78700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.752},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:19.490 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:19.490 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:19.491 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:19.491 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:19.491 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:19.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:19.842 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":950,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:19.855 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:19.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:20.142 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:20.145 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:20.155 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-09-18T14:22:18.162041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003633"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.78700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.752},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:23.157 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:23.458 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:23.476 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-09-18T14:22:22.163205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003506"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.78700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.752},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:23.477 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:23.764 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:23.766 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:23.776 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":486,"stamp":"2024-09-18T14:22:22.163205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003506"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.72599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.78700000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705857,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.53700000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.63900000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.752},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.73199999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705858,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:23.777 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:23.777 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:23.778 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:23.778 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:23.778 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:24.120 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:24.120 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":955,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:24.134 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:24.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:24.429 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:24.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:24.445 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":487,"stamp":"2024-09-18T14:22:24.163826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003575"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.78200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.59499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:27.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:27.744 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:27.747 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:27.758 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-09-18T14:22:26.164366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003514"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.78200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.59499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:27.759 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:28.048 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:28.052 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:28.062 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":488,"stamp":"2024-09-18T14:22:26.164366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003514"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.78200000000000003}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.59499999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.64400000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.84799999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705859,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:28.063 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:28.063 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:28.063 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:28.063 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:28.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:28.423 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:28.423 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":959,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:28.437 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:28.437 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:28.714 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:28.718 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:28.728 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":489,"stamp":"2024-09-18T14:22:28.164868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003386"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.63},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.79100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.65000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.71299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:31.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:32.028 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:32.031 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:32.042 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":490,"stamp":"2024-09-18T14:22:30.165436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003395"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.63},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.79100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.65000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.71299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:32.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:32.329 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:32.333 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:32.343 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-18T14:22:32.165929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003287"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.63},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.79100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.65000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.71299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:32.344 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:32.344 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:32.344 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:32.344 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:32.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:32.684 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:32.685 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":963,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":8,"modified":"2024-09-18T14:19:32.116223+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:32.699 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:32.699 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:32.981 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:32.985 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:32.995 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":491,"stamp":"2024-09-18T14:22:32.165929+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003287"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69199999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.63},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.79100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.65000000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705859,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.71299999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.77100000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705860,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:35.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:36.301 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:36.303 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:36.314 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-18T14:22:36.166984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.73799999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.78800000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:36.316 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:36.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:36.613 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:36.623 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-18T14:22:36.166984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.73799999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.78800000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:36.624 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:36.624 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:36.625 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:36.625 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:36.625 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:36.963 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:36.963 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":967,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-18T14:22:36.167069+0000","services":{"osd":{"daemons":{"summary":"","2":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T14:22:36.976 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:36.977 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:37.259 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:37.262 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:37.272 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":493,"stamp":"2024-09-18T14:22:36.166984+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003157"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.61399999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.56299999999999994}]}]},{"osd":2,"up_from":9,"seq":38654705860,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.69399999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.73799999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.78800000000000003}]}]},{"osd":1,"up_from":0,"seq":38654705862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705861,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:40.274 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:40.571 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:40.574 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:40.584 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-18T14:22:40.168436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003567"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.72999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.64000000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:40.585 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:40.873 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:40.875 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:40.886 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-18T14:22:40.168436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003567"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.72999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.64000000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:40.887 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:40.887 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:40.887 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:40.887 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:40.888 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:41.229 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:41.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":972,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:41.244 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:41.244 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:41.529 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:41.533 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:41.543 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2024-09-18T14:22:40.168436+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003567"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60199999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.72999999999999998}]}]},{"osd":2,"up_from":9,"seq":38654705861,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.54100000000000004}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.46700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.64000000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705862,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:44.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:44.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:44.844 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:44.854 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-18T14:22:44.169473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.59399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.71199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.75900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.79300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:44.855 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:45.143 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:45.154 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-18T14:22:44.169473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.59399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.71199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.75900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.79300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:45.155 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:45.155 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:45.155 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:45.155 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:45.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:45.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:45.502 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":976,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:45.522 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:45.522 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:45.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:45.813 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:45.823 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":497,"stamp":"2024-09-18T14:22:44.169473+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003544"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.59399999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.71199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.64000000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705862,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.75900000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.53300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.79300000000000004}]}]},{"osd":1,"up_from":0,"seq":38654705864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705863,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:48.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:49.121 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:49.135 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-09-18T14:22:48.170651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003667"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.82299999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:49.136 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:49.430 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:49.440 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-09-18T14:22:48.170651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003667"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.82299999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:49.441 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:49.441 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:49.442 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:49.442 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:49.442 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:49.789 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:49.789 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":980,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:49.803 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:49.803 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:50.095 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:50.097 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:50.108 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":499,"stamp":"2024-09-18T14:22:48.170651+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003667"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.82299999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:53.109 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:53.407 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:53.420 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-18T14:22:52.171900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.82299999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:53.421 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:53.697 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:53.701 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:53.711 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":501,"stamp":"2024-09-18T14:22:52.171900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.40799999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.63300000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705863,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.623}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.68999999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.82299999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705864,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:53.712 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:53.712 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:53.712 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:53.712 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:53.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:54.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:54.063 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":984,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:54.086 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:54.086 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:54.379 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:54.382 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:54.392 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2024-09-18T14:22:54.172556+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003617"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.502},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.439}]}]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:57.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:57.689 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:57.693 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:57.703 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-18T14:22:56.173148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.502},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.439}]}]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:57.704 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:57.989 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:57.993 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:58.003 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":503,"stamp":"2024-09-18T14:22:56.173148+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003674"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.502},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60599999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.71699999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.439}]}]},{"osd":2,"up_from":9,"seq":38654705864,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.628}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.81999999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705865,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:22:58.003 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:22:58.004 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:22:58.004 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:22:58.004 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:22:58.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:22:58.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:58.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":989,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:22:58.367 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:22:58.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:22:58.655 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:22:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:22:58.669 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":504,"stamp":"2024-09-18T14:22:58.173641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003547"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.79800000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.60899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.621},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705867,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:01.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:01.967 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:01.970 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:01.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2024-09-18T14:23:00.174169+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003518"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.79800000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.60899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.621},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705867,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:01.985 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:02.272 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:02.274 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:02.289 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2024-09-18T14:23:02.174676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.79800000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.60899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.621},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:02.289 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:23:02.290 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:23:02.290 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:23:02.290 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:23:02.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:23:02.633 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:02.633 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":993,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:23:02.647 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:23:02.648 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:02.933 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:02.936 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:02.947 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":506,"stamp":"2024-09-18T14:23:02.174676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003410"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.69899999999999995}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.79800000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.60899999999999999}]}]},{"osd":2,"up_from":9,"seq":38654705865,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.621},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.47299999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.72999999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:05.948 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:06.247 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:06.250 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:06.261 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-18T14:23:06.175706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003149"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.64700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.751},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.82199999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705867,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:06.262 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:07.383 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:07.389 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:07.404 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-18T14:23:06.175706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003149"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.64700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.751},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.82199999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705867,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:07.405 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:23:07.405 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:23:07.405 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:23:07.409 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:23:07.409 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:23:07.764 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:07.764 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":998,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:23:07.779 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:23:07.779 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:08.066 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:08.069 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:08.079 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":508,"stamp":"2024-09-18T14:23:06.175706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003149"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.56299999999999994}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.64700000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705866,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57799999999999996,"15min":0.59099999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.82299999999999995,"5min":0.86899999999999999,"15min":1.099},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.627,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.67500000000000004,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.67300000000000004,"15min":0.66600000000000004},"min":{"1min":0.53400000000000003,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.879,"5min":0.94399999999999995,"15min":1.1080000000000001},"last":0.751},{"interface":"front","average":{"1min":0.628,"5min":0.64900000000000002,"15min":0.65200000000000002},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.41599999999999998},"max":{"1min":0.79600000000000004,"5min":0.81899999999999995,"15min":0.96699999999999997},"last":0.63100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:22:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.75600000000000001,"15min":0.76300000000000001},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85499999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.755,"5min":0.76900000000000002,"15min":0.77600000000000002},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.85599999999999998,"5min":1.012,"15min":1.222},"last":0.82199999999999995}]}]},{"osd":1,"up_from":0,"seq":38654705868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705867,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:11.081 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:11.375 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:11.389 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-18T14:23:10.176739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003098"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60099999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.501},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.60599999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:11.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:11.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:11.681 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:11.692 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":510,"stamp":"2024-09-18T14:23:10.176739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003098"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60099999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.501},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.60599999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:11.693 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:23:11.693 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:23:11.693 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:23:11.694 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:23:11.694 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:23:12.039 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:12.039 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1002,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:23:12.059 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:23:12.059 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:12.345 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:12.348 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:12.358 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":511,"stamp":"2024-09-18T14:23:12.177222+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003052"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.59699999999999998,"15min":0.60099999999999998},"min":{"1min":0.35199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.88400000000000001,"15min":0.95499999999999996},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.63200000000000001,"15min":0.64000000000000001},"min":{"1min":0.42099999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.84399999999999997,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.60099999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65900000000000003,"15min":0.67900000000000005},"min":{"1min":0.37,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.91400000000000003,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.501,"5min":0.501,"15min":0.376},"max":{"1min":0.88900000000000001,"5min":0.99399999999999999,"15min":1.1499999999999999},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:22:10 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.746,"15min":0.753},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35199999999999998},"max":{"1min":0.90100000000000002,"5min":0.98899999999999999,"15min":1.29},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.70799999999999996,"15min":0.71199999999999997},"min":{"1min":0.434,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.85299999999999998,"5min":1.006,"15min":3.0720000000000001},"last":0.70899999999999996}]}]},{"osd":2,"up_from":9,"seq":38654705867,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.501},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.63700000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.60599999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705868,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:15.360 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:15.654 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:15.656 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:15.666 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-09-18T14:23:14.177745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003069"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.73499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.81599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.879},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.90200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:15.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:15.954 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:15.967 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":512,"stamp":"2024-09-18T14:23:14.177745+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003069"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.73499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.81599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.879},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.90200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:15.968 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:23:15.968 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:23:15.968 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:23:15.968 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:23:15.969 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:23:16.310 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:16.311 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1007,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:23:16.324 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:23:16.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:16.616 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:16.619 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:16.629 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":513,"stamp":"2024-09-18T14:23:16.178272+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003070"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.73499999999999999}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.81599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.879},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.90200000000000002}]}]},{"osd":2,"up_from":9,"seq":38654705868,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.72799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.55800000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76800000000000002}]}]},{"osd":1,"up_from":0,"seq":38654705870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705869,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:19.631 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:19.929 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:19.933 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:19.943 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":514,"stamp":"2024-09-18T14:23:18.178713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003006"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.75600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.82299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:19.944 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:20.237 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:20.241 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:20.251 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-18T14:23:20.179264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003117"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.75600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.82299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:20.252 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:23:20.252 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:23:20.252 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:23:20.252 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:23:20.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:23:20.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:20.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1011,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:23:20.620 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:23:20.620 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:20.912 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:20.914 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:20.925 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2024-09-18T14:23:20.179264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003117"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.75600000000000001}]}]},{"osd":2,"up_from":9,"seq":38654705869,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.82299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.755}]}]},{"osd":1,"up_from":0,"seq":38654705871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705870,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:23.927 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:24.228 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:24.231 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:24.242 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-18T14:23:24.180288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003065"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.754}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.71099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.73699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.80600000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:24.243 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:24.531 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:24.534 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:24.545 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-18T14:23:24.180288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003065"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.754}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.71099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.73699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.80600000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:24.547 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:23:24.547 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:23:24.547 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:23:24.547 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:23:24.548 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:23:24.892 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:24.892 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1015,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:23:24.905 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:23:24.906 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:25.184 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:25.188 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:25.203 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":517,"stamp":"2024-09-18T14:23:24.180288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003065"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.629}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.754}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.71099999999999997}]}]},{"osd":2,"up_from":9,"seq":38654705870,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.60799999999999998}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.73699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.80600000000000005}]}]},{"osd":1,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:28.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:28.503 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:28.506 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:28.516 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-18T14:23:28.181455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003182"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.755},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.78300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76400000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705873,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:28.517 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:28.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:28.810 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:28.821 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-18T14:23:28.181455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003182"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.755},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.78300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76400000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705873,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:28.822 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:23:28.822 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:23:28.822 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:23:28.822 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:23:28.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:23:29.171 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:29.171 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1020,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:23:29.185 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:23:29.186 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:29.482 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:29.485 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:29.500 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":519,"stamp":"2024-09-18T14:23:28.181455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003182"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.755},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.78300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76400000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705873,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:32.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:32.808 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:32.811 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:32.821 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-09-18T14:23:32.182561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003296"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.755},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.78300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76400000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:32.822 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:33.112 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:33.115 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:33.126 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-09-18T14:23:32.182561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003296"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.755},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.78300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76400000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:33.126 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:23:33.127 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "12'1", 'reported_seq': 17, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.051252+0000', 'last_change': '2024-09-18T14:06:51.054231+0000', 'last_active': '2024-09-18T14:06:51.051252+0000', 'last_peered': '2024-09-18T14:06:51.051252+0000', 'last_clean': '2024-09-18T14:06:40.013417+0000', 'last_became_active': '2024-09-18T14:06:51.050888+0000', 'last_became_peered': '2024-09-18T14:06:51.050888+0000', 'last_unstale': '2024-09-18T14:06:51.051252+0000', 'last_undegraded': '2024-09-18T14:06:51.043583+0000', 'last_fullsized': '2024-09-18T14:06:51.051252+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 1, 'log_dups_size': 0, 'ondisk_log_size': 1, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': ['2', '1'], 'object_location_counts': [{'shards': '1,2', 'objects': 1}], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:23:33.127 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:23:33.127 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "14'2", 'reported_seq': 9, 'reported_epoch': 17, 'state': 'active+recovering+degraded', 'last_fresh': '2024-09-18T14:06:51.053996+0000', 'last_change': '2024-09-18T14:06:51.055410+0000', 'last_active': '2024-09-18T14:06:51.053996+0000', 'last_peered': '2024-09-18T14:06:51.053996+0000', 'last_clean': '2024-09-18T14:06:48.862770+0000', 'last_became_active': '2024-09-18T14:06:51.053662+0000', 'last_became_peered': '2024-09-18T14:06:51.053662+0000', 'last_unstale': '2024-09-18T14:06:51.053996+0000', 'last_undegraded': '2024-09-18T14:06:51.046426+0000', 'last_fullsized': '2024-09-18T14:06:51.053996+0000', 'mapping_epoch': 16, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 2, 'log_dups_size': 0, 'ondisk_log_size': 2, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 19, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 1, 'num_objects_missing': 1, 'num_objects_degraded': 0, 'num_objects_misplaced': 2, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 2, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': ['0', '1'], 'object_location_counts': [{'shards': '0,1', 'objects': 1}], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:23:33.127 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:23:33.475 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:33.475 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs degraded","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1024,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":17,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726668410,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+degraded","count":2}],"num_pgs":8,"num_pools":1,"num_objects":2,"data_bytes":19,"bytes_used":609288192,"bytes_avail":192664240128,"bytes_total":193273528320,"misplaced_objects":3,"misplaced_total":4,"misplaced_ratio":0.75},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:23:33.488 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:23:33.489 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:33.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:33.775 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:33.786 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":521,"stamp":"2024-09-18T14:23:32.182561+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595008,"kb_used_data":288,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148672,"statfs":{"total":193273528320,"available":192664240128,"internally_reserved":0,"allocated":294912,"data_stored":88398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003296"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":17,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.051252+0000","last_change":"2024-09-18T14:06:51.054231+0000","last_active":"2024-09-18T14:06:51.051252+0000","last_peered":"2024-09-18T14:06:51.051252+0000","last_clean":"2024-09-18T14:06:40.013417+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:06:51.051252+0000","last_undegraded":"2024-09-18T14:06:51.043583+0000","last_fullsized":"2024-09-18T14:06:51.051252+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":9,"reported_epoch":17,"state":"active+recovering+degraded","last_fresh":"2024-09-18T14:06:51.053996+0000","last_change":"2024-09-18T14:06:51.055410+0000","last_active":"2024-09-18T14:06:51.053996+0000","last_peered":"2024-09-18T14:06:51.053996+0000","last_clean":"2024-09-18T14:06:48.862770+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:06:51.053996+0000","last_undegraded":"2024-09-18T14:06:51.046426+0000","last_fullsized":"2024-09-18T14:06:51.053996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":["0","1"],"object_location_counts":[{"shards":"0,1","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":1,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.58999999999999997}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.71399999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.755},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.78300000000000003}]}]},{"osd":2,"up_from":9,"seq":38654705871,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297504,"kb_used_data":144,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074336,"statfs":{"total":96636764160,"available":96332120064,"internally_reserved":0,"allocated":147456,"data_stored":44199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.56100000000000005}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.623}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.76400000000000001}]}]},{"osd":1,"up_from":0,"seq":38654705874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705872,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:36.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:23:37.079 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:23:37.081 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:23:37.097 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":523,"stamp":"2024-09-18T14:23:36.183739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595000,"kb_used_data":296,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148680,"statfs":{"total":193273528320,"available":192664248320,"internally_reserved":0,"allocated":303104,"data_stored":88436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":-1,"num_objects_missing":-1,"num_objects_degraded":0,"num_objects_misplaced":-3,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003451"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":12,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.050935+0000","last_change":"2024-09-18T14:06:51.050935+0000","last_active":"2024-09-18T14:06:51.050935+0000","last_peered":"2024-09-18T14:06:51.050935+0000","last_clean":"2024-09-18T14:06:51.050935+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:06:51.050935+0000","last_undegraded":"2024-09-18T14:06:51.050935+0000","last_fullsized":"2024-09-18T14:06:51.050935+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:56:57.481432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052029+0000","last_change":"2024-09-18T14:06:51.052029+0000","last_active":"2024-09-18T14:06:51.052029+0000","last_peered":"2024-09-18T14:06:51.052029+0000","last_clean":"2024-09-18T14:06:51.052029+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:06:51.052029+0000","last_undegraded":"2024-09-18T14:06:51.052029+0000","last_fullsized":"2024-09-18T14:06:51.052029+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:05:46.667399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":18,"reported_epoch":16,"state":"active+clean","last_fresh":"2024-09-18T14:06:50.039687+0000","last_change":"2024-09-18T14:06:42.016777+0000","last_active":"2024-09-18T14:06:50.039687+0000","last_peered":"2024-09-18T14:06:50.039687+0000","last_clean":"2024-09-18T14:06:50.039687+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:06:50.039687+0000","last_undegraded":"2024-09-18T14:06:50.039687+0000","last_fullsized":"2024-09-18T14:06:50.039687+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:57:19.946109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051969+0000","last_change":"2024-09-18T14:06:51.051969+0000","last_active":"2024-09-18T14:06:51.051969+0000","last_peered":"2024-09-18T14:06:51.051969+0000","last_clean":"2024-09-18T14:06:51.051969+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:06:51.051969+0000","last_undegraded":"2024-09-18T14:06:51.051969+0000","last_fullsized":"2024-09-18T14:06:51.051969+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:21:31.751443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"12'1","reported_seq":22,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:23:29.940985+0000","last_change":"2024-09-18T14:23:29.940985+0000","last_active":"2024-09-18T14:23:29.940985+0000","last_peered":"2024-09-18T14:23:29.940985+0000","last_clean":"2024-09-18T14:23:29.940985+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:23:29.940985+0000","last_undegraded":"2024-09-18T14:23:29.940985+0000","last_fullsized":"2024-09-18T14:23:29.940985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:48:44.428203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"14'2","reported_seq":15,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:23:29.807419+0000","last_change":"2024-09-18T14:23:29.807419+0000","last_active":"2024-09-18T14:23:29.807419+0000","last_peered":"2024-09-18T14:23:29.807419+0000","last_clean":"2024-09-18T14:23:29.807419+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:23:29.807419+0000","last_undegraded":"2024-09-18T14:23:29.807419+0000","last_fullsized":"2024-09-18T14:23:29.807419+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:03:53.116190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":27,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.052642+0000","last_change":"2024-09-18T14:06:51.052642+0000","last_active":"2024-09-18T14:06:51.052642+0000","last_peered":"2024-09-18T14:06:51.052642+0000","last_clean":"2024-09-18T14:06:51.052642+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:06:51.052642+0000","last_undegraded":"2024-09-18T14:06:51.052642+0000","last_fullsized":"2024-09-18T14:06:51.052642+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:04:54.259579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":9,"reported_epoch":17,"state":"active+clean","last_fresh":"2024-09-18T14:06:51.051263+0000","last_change":"2024-09-18T14:06:51.051263+0000","last_active":"2024-09-18T14:06:51.051263+0000","last_peered":"2024-09-18T14:06:51.051263+0000","last_clean":"2024-09-18T14:06:51.051263+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:06:51.051263+0000","last_undegraded":"2024-09-18T14:06:51.051263+0000","last_fullsized":"2024-09-18T14:06:51.051263+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:58.727890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":76,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":9,"seq":38654705872,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":151552,"data_stored":44218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.59799999999999998,"15min":0.59899999999999998},"min":{"1min":0.41299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.95499999999999996},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63700000000000001,"15min":0.64100000000000001},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.78200000000000003,"5min":0.90800000000000003,"15min":0.97399999999999998},"last":0.51100000000000001}]},{"osd":1,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.65200000000000002,"15min":0.67600000000000005},"min":{"1min":0.439,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81299999999999994,"5min":0.93700000000000006,"15min":1.0760000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70799999999999996,"15min":0.69699999999999995},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.376},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":1.1499999999999999},"last":0.63500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:23:10 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.73999999999999999,"15min":0.75},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.88400000000000001,"5min":0.94099999999999995,"15min":1.29},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.69099999999999995,"15min":0.70699999999999996},"min":{"1min":0.439,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.79900000000000004,"5min":0.95199999999999996,"15min":3.0720000000000001},"last":0.81399999999999995}]}]},{"osd":2,"up_from":9,"seq":38654705872,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297500,"kb_used_data":148,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074340,"statfs":{"total":96636764160,"available":96332124160,"internally_reserved":0,"allocated":151552,"data_stored":44218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.57899999999999996,"15min":0.58799999999999997},"min":{"1min":0.44400000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":1.099},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.623,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.40500000000000003,"15min":0.34200000000000003},"max":{"1min":0.77400000000000002,"5min":0.95399999999999996,"15min":1.0780000000000001},"last":0.65200000000000002}]},{"osd":1,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.67100000000000004,"15min":0.66500000000000004},"min":{"1min":0.42099999999999999,"5min":0.39200000000000002,"15min":0.33200000000000002},"max":{"1min":0.84699999999999998,"5min":0.879,"15min":1.1080000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64400000000000002,"15min":0.65400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.41599999999999998},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.72099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:23:05 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.752,"15min":0.76000000000000001},"min":{"1min":0.61399999999999999,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.85699999999999998,"5min":0.93100000000000005,"15min":1.085},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.77100000000000002,"15min":0.77600000000000002},"min":{"1min":0.46700000000000003,"5min":0.46000000000000002,"15min":0.438},"max":{"1min":0.94999999999999996,"5min":1.012,"15min":1.1359999999999999},"last":0.60599999999999998}]}]},{"osd":1,"up_from":0,"seq":38654705874,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":0,"seq":38654705873,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:23:37.098 INFO:tasks.osd_recovery.ceph_manager:clean! 2024-09-18T14:23:37.098 INFO:tasks.osd_recovery:rados -p rbd bench 20 write -b 1 --no-cleanup 2024-09-18T14:23:37.098 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd bench 20 write -b 1 --no-cleanup 2024-09-18T14:23:37.140 INFO:teuthology.orchestra.run.smithi016.stdout:hints = 1 2024-09-18T14:23:37.140 INFO:teuthology.orchestra.run.smithi016.stdout:Maintaining 16 concurrent writes of 1 bytes to objects of size 1 for up to 20 seconds or 0 objects 2024-09-18T14:23:37.140 INFO:teuthology.orchestra.run.smithi016.stdout:Object prefix: benchmark_data_smithi016_52446 2024-09-18T14:23:37.140 INFO:teuthology.orchestra.run.smithi016.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:23:37.140 INFO:teuthology.orchestra.run.smithi016.stdout: 0 0 0 0 0 0 - 0 2024-09-18T14:23:37.173 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:37.167+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed38e00 msgr2=0x56356ed3b2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:37.180 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:37.175+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x56356ed3de60 msgr2=0x56356ed40310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:38.105 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:38.099+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:38.114 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:38.107+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed38e00 msgr2=0x7fe04803c9c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T14:23:38.116 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:38.111+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T14:23:38.140 INFO:teuthology.orchestra.run.smithi016.stdout: 1 16 3640 3624 0.00345528 0.00345612 0.00525147 0.00440271 2024-09-18T14:23:38.217 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:38.211+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x56356ed3de60 msgr2=0x7fe04803cf80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:38.845 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:38.839+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x56356ed3de60 msgr2=0x7fe04803ce40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:39.140 INFO:teuthology.orchestra.run.smithi016.stdout: 2 16 6810 6794 0.0032391 0.00302315 0.00287964 0.00469891 2024-09-18T14:23:39.410 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:39.403+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe04803d780 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T14:23:39.583 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:39.579+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x56356ed38e00 msgr2=0x7fe04803ee40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:40.086 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:40.079+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:40.140 INFO:teuthology.orchestra.run.smithi016.stdout: 3 16 8999 8983 0.00285523 0.00208759 0.00803094 0.00533396 2024-09-18T14:23:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:40.687+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:41.140 INFO:teuthology.orchestra.run.smithi016.stdout: 4 16 11522 11506 0.00274291 0.00240612 0.00615962 0.00555358 2024-09-18T14:23:42.140 INFO:teuthology.orchestra.run.smithi016.stdout: 5 16 13782 13766 0.00262536 0.0021553 0.00394726 0.00580299 2024-09-18T14:23:42.197 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:42.191+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe04803d680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:43.140 INFO:teuthology.orchestra.run.smithi016.stdout: 6 16 15861 15845 0.00251822 0.00198269 0.00596507 0.00605231 2024-09-18T14:23:43.245 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:43.239+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:43.482 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:43.475+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed38e00 msgr2=0x7fe04803e030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:44.140 INFO:teuthology.orchestra.run.smithi016.stdout: 7 16 17688 17672 0.00240737 0.00174236 0.00567671 0.00633139 2024-09-18T14:23:45.038 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:45.031+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe04803c900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T14:23:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:45.071+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed38e00 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:45.123 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:45.119+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T14:23:45.140 INFO:teuthology.orchestra.run.smithi016.stdout: 8 16 19548 19532 0.00232816 0.00177383 0.00412864 0.00654584 2024-09-18T14:23:45.552 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:45.547+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed38e00 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:45.932 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:45.927+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:45.988 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:45.983+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:46.140 INFO:teuthology.orchestra.run.smithi016.stdout: 9 16 21272 21256 0.00225214 0.00164413 0.00768646 0.00676759 2024-09-18T14:23:46.555 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:46.551+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe04803f110 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:46.715+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:46.721 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:46.715+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:46.807 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:46.803+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:46.831+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:47.142 INFO:teuthology.orchestra.run.smithi016.stdout: 10 16 22927 22911 0.00218451 0.00157833 0.00916286 0.00697732 2024-09-18T14:23:47.316 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:47.311+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T14:23:47.797 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:47.791+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T14:23:48.142 INFO:teuthology.orchestra.run.smithi016.stdout: 11 16 24484 24468 0.0021209 0.00148487 0.0164326 0.00718773 2024-09-18T14:23:48.243 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:48.239+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:48.998 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:48.991+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:49.046 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:49.039+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe04803f110 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:49.142 INFO:teuthology.orchestra.run.smithi016.stdout: 12 16 25979 25963 0.00206297 0.00142574 0.00852039 0.00738878 2024-09-18T14:23:49.449 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:49.443+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:49.914 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:49.907+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:50.142 INFO:teuthology.orchestra.run.smithi016.stdout: 13 16 27414 27398 0.00200954 0.00136852 0.0150033 0.00758653 2024-09-18T14:23:51.142 INFO:teuthology.orchestra.run.smithi016.stdout: 14 16 28796 28780 0.00196014 0.00131798 0.0140155 0.00777774 2024-09-18T14:23:51.171 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:51.162+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:51.822 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:51.814+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe04803f110 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:51.890 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:51.882+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:52.142 INFO:teuthology.orchestra.run.smithi016.stdout: 15 16 30121 30105 0.00191371 0.00126362 0.0291871 0.00796552 2024-09-18T14:23:52.450 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:52.442+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe04803f110 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:53.145 INFO:teuthology.orchestra.run.smithi016.stdout: 16 16 31406 31390 0.00187069 0.00122547 0.0203763 0.00815089 2024-09-18T14:23:53.152 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:53.146+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe04803f110 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:54.145 INFO:teuthology.orchestra.run.smithi016.stdout: 17 16 32640 32624 0.00182987 0.00117683 0.0123182 0.00833171 2024-09-18T14:23:54.565 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:54.558+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2024-09-18T14:23:54.722 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:54.714+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:54.880 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:54.874+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:54.894+0000 7fe064afa700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6816/22310,v1:172.21.15.16:6817/22310] conn(0x56356ed3de60 msgr2=0x7fe048043df0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:55.142 INFO:teuthology.orchestra.run.smithi016.stdout: 18 16 33853 33837 0.00179247 0.00115681 0.0213149 0.00850539 2024-09-18T14:23:55.511 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:55.506+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe04803f110 msgr2=0x7fe0480438b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:56.142 INFO:teuthology.orchestra.run.smithi016.stdout: 19 16 35049 35033 0.00175816 0.00114059 0.00822412 0.00867217 2024-09-18T14:23:56.185 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:23:56.178+0000 7fe0652fb700 0 -- 172.21.15.16:0/2142865525 >> [v2:172.21.15.16:6800/22295,v1:172.21.15.16:6801/22295] conn(0x7fe048039f10 msgr2=0x7fe04803c2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2024-09-18T14:23:57.142 INFO:teuthology.orchestra.run.smithi016.stdout:2024-09-18T14:23:57.140041+0000 min lat: 0.00111626 max lat: 0.218555 avg lat: 0.00883377 2024-09-18T14:23:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-18T14:23:57.143 INFO:teuthology.orchestra.run.smithi016.stdout: 20 16 36214 36198 0.0017258 0.00111103 0.00777924 0.00883377 2024-09-18T14:23:58.143 INFO:teuthology.orchestra.run.smithi016.stdout:Total time run: 20.0057 2024-09-18T14:23:58.143 INFO:teuthology.orchestra.run.smithi016.stdout:Total writes made: 36214 2024-09-18T14:23:58.143 INFO:teuthology.orchestra.run.smithi016.stdout:Write size: 1 2024-09-18T14:23:58.143 INFO:teuthology.orchestra.run.smithi016.stdout:Object size: 1 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Bandwidth (MB/sec): 0.00172633 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Stddev Bandwidth: 0.00064088 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Max bandwidth (MB/sec): 0.00345612 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Min bandwidth (MB/sec): 0.00111103 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Average IOPS: 1810 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Stddev IOPS: 672.011 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Max IOPS: 3624 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Min IOPS: 1165 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Average Latency(s): 0.00883539 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Stddev Latency(s): 0.00648656 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Max latency(s): 0.218555 2024-09-18T14:23:58.144 INFO:teuthology.orchestra.run.smithi016.stdout:Min latency(s): 0.00111626 2024-09-18T14:23:58.156 INFO:tasks.osd_recovery:rados -p rbd put foo.1 /etc/passwd 2024-09-18T14:23:58.156 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.1 /etc/passwd 2024-09-18T14:23:58.210 INFO:tasks.osd_recovery:rados -p rbd put foo.2 /etc/passwd 2024-09-18T14:23:58.210 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.2 /etc/passwd 2024-09-18T14:23:58.265 INFO:tasks.osd_recovery:rados -p rbd put foo.3 /etc/passwd 2024-09-18T14:23:58.265 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.3 /etc/passwd 2024-09-18T14:23:58.318 INFO:tasks.osd_recovery:rados -p rbd put foo.4 /etc/passwd 2024-09-18T14:23:58.318 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.4 /etc/passwd 2024-09-18T14:23:58.366 INFO:tasks.osd_recovery:rados -p rbd put foo.5 /etc/passwd 2024-09-18T14:23:58.367 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.5 /etc/passwd 2024-09-18T14:23:58.417 INFO:tasks.osd_recovery:rados -p rbd put foo.6 /etc/passwd 2024-09-18T14:23:58.418 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.6 /etc/passwd 2024-09-18T14:23:58.467 INFO:tasks.osd_recovery:rados -p rbd put foo.7 /etc/passwd 2024-09-18T14:23:58.467 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.7 /etc/passwd 2024-09-18T14:23:58.519 INFO:tasks.osd_recovery:rados -p rbd put foo.8 /etc/passwd 2024-09-18T14:23:58.519 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.8 /etc/passwd 2024-09-18T14:23:58.574 INFO:tasks.osd_recovery:rados -p rbd put foo.9 /etc/passwd 2024-09-18T14:23:58.574 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.9 /etc/passwd 2024-09-18T14:23:58.629 INFO:tasks.osd_recovery:rados -p rbd put foo.10 /etc/passwd 2024-09-18T14:23:58.629 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.10 /etc/passwd 2024-09-18T14:23:58.680 INFO:tasks.osd_recovery:rados -p rbd put foo.11 /etc/passwd 2024-09-18T14:23:58.680 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.11 /etc/passwd 2024-09-18T14:23:58.734 INFO:tasks.osd_recovery:rados -p rbd put foo.12 /etc/passwd 2024-09-18T14:23:58.734 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.12 /etc/passwd 2024-09-18T14:23:58.790 INFO:tasks.osd_recovery:rados -p rbd put foo.13 /etc/passwd 2024-09-18T14:23:58.790 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.13 /etc/passwd 2024-09-18T14:23:59.051 INFO:tasks.osd_recovery:rados -p rbd put foo.14 /etc/passwd 2024-09-18T14:23:59.052 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.14 /etc/passwd 2024-09-18T14:23:59.108 INFO:tasks.osd_recovery:rados -p rbd put foo.15 /etc/passwd 2024-09-18T14:23:59.109 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.15 /etc/passwd 2024-09-18T14:23:59.161 INFO:tasks.osd_recovery:rados -p rbd put foo.16 /etc/passwd 2024-09-18T14:23:59.162 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.16 /etc/passwd 2024-09-18T14:23:59.214 INFO:tasks.osd_recovery:rados -p rbd put foo.17 /etc/passwd 2024-09-18T14:23:59.214 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.17 /etc/passwd 2024-09-18T14:23:59.266 INFO:tasks.osd_recovery:rados -p rbd put foo.18 /etc/passwd 2024-09-18T14:23:59.266 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.18 /etc/passwd 2024-09-18T14:23:59.317 INFO:tasks.osd_recovery:rados -p rbd put foo.19 /etc/passwd 2024-09-18T14:23:59.373 DEBUG:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd put foo.19 /etc/passwd 2024-09-18T14:23:59.429 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 0 1 2024-09-18T14:23:59.738 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-18T14:23:59.730+0000 7f755a98d700 -1 mon.a@0(leader).osd e17 definitely_dead 0 2024-09-18T14:24:00.584 INFO:teuthology.orchestra.run.smithi016.stderr:marked in osd.0. marked in osd.1. 2024-09-18T14:24:00.597 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 2 3 2024-09-18T14:24:00.915 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-18T14:24:00.910+0000 7f755a98d700 -1 mon.a@0(leader).osd e18 definitely_dead 0 2024-09-18T14:24:01.600 INFO:teuthology.orchestra.run.smithi016.stderr:marked out osd.2. marked out osd.3. 2024-09-18T14:24:11.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T14:24:11.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T14:24:11.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-18T14:24:11.613 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-18T14:24:11.770 INFO:teuthology.orchestra.run.smithi016.stdout:38654705880 2024-09-18T14:24:11.771 DEBUG:teuthology.parallel:result is None 2024-09-18T14:24:11.786 INFO:teuthology.orchestra.run.smithi016.stdout:38654705882 2024-09-18T14:24:11.786 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:24:11.802 INFO:teuthology.orchestra.run.smithi016.stdout:38654705880 2024-09-18T14:24:11.803 DEBUG:teuthology.parallel:result is None 2024-09-18T14:24:11.804 INFO:teuthology.orchestra.run.smithi016.stdout:38654705881 2024-09-18T14:24:11.804 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:24:12.090 INFO:teuthology.orchestra.run.smithi016.stdout:38654705881 2024-09-18T14:24:12.106 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705882 got 38654705881 for osd.1 2024-09-18T14:24:12.182 INFO:teuthology.orchestra.run.smithi016.stdout:38654705880 2024-09-18T14:24:12.196 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705881 got 38654705880 for osd.0 2024-09-18T14:24:13.107 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:24:13.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:24:13.439 INFO:teuthology.orchestra.run.smithi016.stdout:38654705883 2024-09-18T14:24:13.455 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705882 got 38654705883 for osd.1 2024-09-18T14:24:13.455 DEBUG:teuthology.parallel:result is None 2024-09-18T14:24:13.489 INFO:teuthology.orchestra.run.smithi016.stdout:38654705881 2024-09-18T14:24:13.508 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705881 got 38654705881 for osd.0 2024-09-18T14:24:13.508 DEBUG:teuthology.parallel:result is None 2024-09-18T14:24:23.508 INFO:tasks.osd_recovery.ceph_manager:waiting for peering to complete 2024-09-18T14:24:23.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:23.842 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:23.845 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:23.857 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2024-09-18T14:24:22.192895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":39377,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":683,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":599212,"kb_used_data":348,"kb_used_omap":3,"kb_used_meta":598844,"kb_avail":188144468,"statfs":{"total":193273528320,"available":192659935232,"internally_reserved":0,"allocated":356352,"data_stored":115740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":613216944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-932,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003059"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4616,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.298944+0000","last_change":"2024-09-18T14:24:04.300760+0000","last_active":"2024-09-18T14:24:04.298944+0000","last_peered":"2024-09-18T14:24:04.298944+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:04.294785+0000","last_became_peered":"2024-09-18T14:24:04.294785+0000","last_unstale":"2024-09-18T14:24:04.298944+0000","last_undegraded":"2024-09-18T14:24:03.746756+0000","last_fullsized":"2024-09-18T14:24:03.597698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:11:51.954241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4589,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.196021+0000","last_change":"2024-09-18T14:24:03.196021+0000","last_active":"2024-09-18T14:24:03.196021+0000","last_peered":"2024-09-18T14:24:03.196021+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:03.185799+0000","last_became_peered":"2024-09-18T14:24:03.185799+0000","last_unstale":"2024-09-18T14:24:03.196021+0000","last_undegraded":"2024-09-18T14:24:02.968820+0000","last_fullsized":"2024-09-18T14:24:02.598535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:22:05.974865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4482,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.312093+0000","last_change":"2024-09-18T14:24:04.313567+0000","last_active":"2024-09-18T14:24:04.312093+0000","last_peered":"2024-09-18T14:24:04.312093+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:04.308184+0000","last_became_peered":"2024-09-18T14:24:04.308184+0000","last_unstale":"2024-09-18T14:24:04.312093+0000","last_undegraded":"2024-09-18T14:24:03.738484+0000","last_fullsized":"2024-09-18T14:24:03.597525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:08:54.672615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":6588,"reported_epoch":21,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:24:18.097396+0000","last_change":"2024-09-18T14:24:03.195733+0000","last_active":"2024-09-18T14:24:18.097396+0000","last_peered":"2024-09-18T14:24:18.097396+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:03.165002+0000","last_became_peered":"2024-09-18T14:24:03.165002+0000","last_unstale":"2024-09-18T14:24:18.097396+0000","last_undegraded":"2024-09-18T14:24:18.097396+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:29:38.690657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7644,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3822},{"shards":"3","objects":681}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4514,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.213972+0000","last_change":"2024-09-18T14:24:03.213972+0000","last_active":"2024-09-18T14:24:03.213972+0000","last_peered":"2024-09-18T14:24:03.213972+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:03.206328+0000","last_became_peered":"2024-09-18T14:24:03.206328+0000","last_unstale":"2024-09-18T14:24:03.213972+0000","last_undegraded":"2024-09-18T14:24:02.948624+0000","last_fullsized":"2024-09-18T14:24:02.596557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:09:07.111338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4660,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.218843+0000","last_change":"2024-09-18T14:24:03.218843+0000","last_active":"2024-09-18T14:24:03.218843+0000","last_peered":"2024-09-18T14:24:03.218843+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:03.209404+0000","last_became_peered":"2024-09-18T14:24:03.209404+0000","last_unstale":"2024-09-18T14:24:03.218843+0000","last_undegraded":"2024-09-18T14:24:02.908616+0000","last_fullsized":"2024-09-18T14:24:02.596585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:24:21.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4636,"num_objects_misplaced":4636,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4636}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4531,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.219319+0000","last_change":"2024-09-18T14:24:03.219319+0000","last_active":"2024-09-18T14:24:03.219319+0000","last_peered":"2024-09-18T14:24:03.219319+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:03.209636+0000","last_became_peered":"2024-09-18T14:24:03.209636+0000","last_unstale":"2024-09-18T14:24:03.219319+0000","last_undegraded":"2024-09-18T14:24:02.903337+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:02:48.059016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4551,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.186314+0000","last_change":"2024-09-18T14:24:03.186314+0000","last_active":"2024-09-18T14:24:03.186314+0000","last_peered":"2024-09-18T14:24:03.186314+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:03.177160+0000","last_became_peered":"2024-09-18T14:24:03.177160+0000","last_unstale":"2024-09-18T14:24:03.186314+0000","last_undegraded":"2024-09-18T14:24:02.967240+0000","last_fullsized":"2024-09-18T14:24:02.596829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:45.060182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":39377,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":683,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":498494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":38654705882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":38654705882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705885,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":180224,"data_stored":58159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.376},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297524,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074316,"statfs":{"total":96636764160,"available":96332099584,"internally_reserved":0,"allocated":176128,"data_stored":57581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.60399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:23.858 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:24.152 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:24.154 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:24.163 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":550,"stamp":"2024-09-18T14:24:22.192895+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":39377,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":683,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":599212,"kb_used_data":348,"kb_used_omap":3,"kb_used_meta":598844,"kb_avail":188144468,"statfs":{"total":193273528320,"available":192659935232,"internally_reserved":0,"allocated":356352,"data_stored":115740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":613216944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-932,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003059"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4616,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.298944+0000","last_change":"2024-09-18T14:24:04.300760+0000","last_active":"2024-09-18T14:24:04.298944+0000","last_peered":"2024-09-18T14:24:04.298944+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:04.294785+0000","last_became_peered":"2024-09-18T14:24:04.294785+0000","last_unstale":"2024-09-18T14:24:04.298944+0000","last_undegraded":"2024-09-18T14:24:03.746756+0000","last_fullsized":"2024-09-18T14:24:03.597698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:11:51.954241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4589,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.196021+0000","last_change":"2024-09-18T14:24:03.196021+0000","last_active":"2024-09-18T14:24:03.196021+0000","last_peered":"2024-09-18T14:24:03.196021+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:03.185799+0000","last_became_peered":"2024-09-18T14:24:03.185799+0000","last_unstale":"2024-09-18T14:24:03.196021+0000","last_undegraded":"2024-09-18T14:24:02.968820+0000","last_fullsized":"2024-09-18T14:24:02.598535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:22:05.974865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4482,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.312093+0000","last_change":"2024-09-18T14:24:04.313567+0000","last_active":"2024-09-18T14:24:04.312093+0000","last_peered":"2024-09-18T14:24:04.312093+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:04.308184+0000","last_became_peered":"2024-09-18T14:24:04.308184+0000","last_unstale":"2024-09-18T14:24:04.312093+0000","last_undegraded":"2024-09-18T14:24:03.738484+0000","last_fullsized":"2024-09-18T14:24:03.597525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:08:54.672615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":6588,"reported_epoch":21,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:24:18.097396+0000","last_change":"2024-09-18T14:24:03.195733+0000","last_active":"2024-09-18T14:24:18.097396+0000","last_peered":"2024-09-18T14:24:18.097396+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:03.165002+0000","last_became_peered":"2024-09-18T14:24:03.165002+0000","last_unstale":"2024-09-18T14:24:18.097396+0000","last_undegraded":"2024-09-18T14:24:18.097396+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:29:38.690657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7644,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3822},{"shards":"3","objects":681}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4514,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.213972+0000","last_change":"2024-09-18T14:24:03.213972+0000","last_active":"2024-09-18T14:24:03.213972+0000","last_peered":"2024-09-18T14:24:03.213972+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:03.206328+0000","last_became_peered":"2024-09-18T14:24:03.206328+0000","last_unstale":"2024-09-18T14:24:03.213972+0000","last_undegraded":"2024-09-18T14:24:02.948624+0000","last_fullsized":"2024-09-18T14:24:02.596557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:09:07.111338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4660,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.218843+0000","last_change":"2024-09-18T14:24:03.218843+0000","last_active":"2024-09-18T14:24:03.218843+0000","last_peered":"2024-09-18T14:24:03.218843+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:03.209404+0000","last_became_peered":"2024-09-18T14:24:03.209404+0000","last_unstale":"2024-09-18T14:24:03.218843+0000","last_undegraded":"2024-09-18T14:24:02.908616+0000","last_fullsized":"2024-09-18T14:24:02.596585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:24:21.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4636,"num_objects_misplaced":4636,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4636}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4531,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.219319+0000","last_change":"2024-09-18T14:24:03.219319+0000","last_active":"2024-09-18T14:24:03.219319+0000","last_peered":"2024-09-18T14:24:03.219319+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:03.209636+0000","last_became_peered":"2024-09-18T14:24:03.209636+0000","last_unstale":"2024-09-18T14:24:03.219319+0000","last_undegraded":"2024-09-18T14:24:02.903337+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:02:48.059016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4551,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.186314+0000","last_change":"2024-09-18T14:24:03.186314+0000","last_active":"2024-09-18T14:24:03.186314+0000","last_peered":"2024-09-18T14:24:03.186314+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:03.177160+0000","last_became_peered":"2024-09-18T14:24:03.177160+0000","last_unstale":"2024-09-18T14:24:03.186314+0000","last_undegraded":"2024-09-18T14:24:02.967240+0000","last_fullsized":"2024-09-18T14:24:02.596829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:45.060182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":39377,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":683,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":498494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":38654705882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":38654705882,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705885,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":180224,"data_stored":58159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.376},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705883,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297524,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074316,"statfs":{"total":96636764160,"available":96332099584,"internally_reserved":0,"allocated":176128,"data_stored":57581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.60399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":1.0800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:24.165 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:24:24.537 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:24.537 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 31733/72472 objects degraded (43.787%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1075,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":21,"num_osds":4,"num_up_osds":4,"osd_up_since":1726668396,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":613593088,"bytes_avail":192659935232,"bytes_total":193273528320,"degraded_objects":31733,"degraded_total":72472,"degraded_ratio":0.43786565846119879,"misplaced_objects":39377,"misplaced_total":72472,"misplaced_ratio":0.54334087647643226,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":466,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:24:24.552 INFO:tasks.osd_recovery.ceph_manager:{'fsid': '946c9667-3cd7-48a9-8c9c-fd6b4af766c9', 'health': {'status': 'HEALTH_WARN', 'checks': {'PG_DEGRADED': {'severity': 'HEALTH_WARN', 'summary': {'message': 'Degraded data redundancy: 31733/72472 objects degraded (43.787%), 7 pgs degraded', 'count': 7}, 'muted': False}}, 'mutes': []}, 'election_epoch': 6, 'quorum': [0, 1, 2], 'quorum_names': ['a', 'b', 'c'], 'quorum_age': 1075, 'monmap': {'epoch': 1, 'min_mon_release_name': 'quincy', 'num_mons': 3}, 'osdmap': {'epoch': 21, 'num_osds': 4, 'num_up_osds': 4, 'osd_up_since': 1726668396, 'num_in_osds': 2, 'osd_in_since': 1726669441, 'num_remapped_pgs': 8}, 'pgmap': {'pgs_by_state': [{'state_name': 'active+recovery_wait+undersized+degraded+remapped', 'count': 7}, {'state_name': 'active+recovering+undersized+remapped', 'count': 1}], 'num_pgs': 8, 'num_pools': 1, 'num_objects': 36236, 'data_bytes': 285461, 'bytes_used': 613593088, 'bytes_avail': 192659935232, 'bytes_total': 193273528320, 'degraded_objects': 31733, 'degraded_total': 72472, 'degraded_ratio': 0.4378656584611988, 'misplaced_objects': 39377, 'misplaced_total': 72472, 'misplaced_ratio': 0.5433408764764323, 'recovering_objects_per_sec': 38, 'recovering_bytes_per_sec': 0, 'recovering_keys_per_sec': 0, 'num_objects_recovered': 466, 'num_bytes_recovered': 0, 'num_keys_recovered': 0}, 'fsmap': {'epoch': 1, 'by_rank': [], 'up:standby': 0}, 'mgrmap': {'available': True, 'num_standbys': 0, 'modules': ['iostat', 'nfs', 'restful'], 'services': {}}, 'servicemap': {'epoch': 10, 'modified': '2024-09-18T14:22:38.167822+0000', 'services': {}}, 'progress_events': {}} 2024-09-18T14:24:24.553 INFO:tasks.osd_recovery.ceph_manager:active! 2024-09-18T14:24:24.553 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:24.878 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:24.881 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:24.891 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-09-18T14:24:24.193445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":38875,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603372,"kb_used_data":348,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140308,"statfs":{"total":193273528320,"available":192655675392,"internally_reserved":0,"allocated":356352,"data_stored":115740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4616,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.298944+0000","last_change":"2024-09-18T14:24:04.300760+0000","last_active":"2024-09-18T14:24:04.298944+0000","last_peered":"2024-09-18T14:24:04.298944+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:04.294785+0000","last_became_peered":"2024-09-18T14:24:04.294785+0000","last_unstale":"2024-09-18T14:24:04.298944+0000","last_undegraded":"2024-09-18T14:24:03.746756+0000","last_fullsized":"2024-09-18T14:24:03.597698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:11:51.954241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4589,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.196021+0000","last_change":"2024-09-18T14:24:03.196021+0000","last_active":"2024-09-18T14:24:03.196021+0000","last_peered":"2024-09-18T14:24:03.196021+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:03.185799+0000","last_became_peered":"2024-09-18T14:24:03.185799+0000","last_unstale":"2024-09-18T14:24:03.196021+0000","last_undegraded":"2024-09-18T14:24:02.968820+0000","last_fullsized":"2024-09-18T14:24:02.598535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:22:05.974865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4482,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.312093+0000","last_change":"2024-09-18T14:24:04.313567+0000","last_active":"2024-09-18T14:24:04.312093+0000","last_peered":"2024-09-18T14:24:04.312093+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:04.308184+0000","last_became_peered":"2024-09-18T14:24:04.308184+0000","last_unstale":"2024-09-18T14:24:04.312093+0000","last_undegraded":"2024-09-18T14:24:03.738484+0000","last_fullsized":"2024-09-18T14:24:03.597525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:08:54.672615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":7341,"reported_epoch":21,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:24:23.100835+0000","last_change":"2024-09-18T14:24:03.195733+0000","last_active":"2024-09-18T14:24:23.100835+0000","last_peered":"2024-09-18T14:24:23.100835+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:03.165002+0000","last_became_peered":"2024-09-18T14:24:03.165002+0000","last_unstale":"2024-09-18T14:24:23.100835+0000","last_undegraded":"2024-09-18T14:24:23.100835+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:29:38.690657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7142,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3571},{"shards":"3","objects":932}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4514,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.213972+0000","last_change":"2024-09-18T14:24:03.213972+0000","last_active":"2024-09-18T14:24:03.213972+0000","last_peered":"2024-09-18T14:24:03.213972+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:03.206328+0000","last_became_peered":"2024-09-18T14:24:03.206328+0000","last_unstale":"2024-09-18T14:24:03.213972+0000","last_undegraded":"2024-09-18T14:24:02.948624+0000","last_fullsized":"2024-09-18T14:24:02.596557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:09:07.111338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4660,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.218843+0000","last_change":"2024-09-18T14:24:03.218843+0000","last_active":"2024-09-18T14:24:03.218843+0000","last_peered":"2024-09-18T14:24:03.218843+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:03.209404+0000","last_became_peered":"2024-09-18T14:24:03.209404+0000","last_unstale":"2024-09-18T14:24:03.218843+0000","last_undegraded":"2024-09-18T14:24:02.908616+0000","last_fullsized":"2024-09-18T14:24:02.596585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:24:21.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4636,"num_objects_misplaced":4636,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4636}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4531,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.219319+0000","last_change":"2024-09-18T14:24:03.219319+0000","last_active":"2024-09-18T14:24:03.219319+0000","last_peered":"2024-09-18T14:24:03.219319+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:03.209636+0000","last_became_peered":"2024-09-18T14:24:03.209636+0000","last_unstale":"2024-09-18T14:24:03.219319+0000","last_undegraded":"2024-09-18T14:24:02.903337+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:02:48.059016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4551,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.186314+0000","last_change":"2024-09-18T14:24:03.186314+0000","last_active":"2024-09-18T14:24:03.186314+0000","last_peered":"2024-09-18T14:24:03.186314+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:03.177160+0000","last_became_peered":"2024-09-18T14:24:03.177160+0000","last_unstale":"2024-09-18T14:24:03.186314+0000","last_undegraded":"2024-09-18T14:24:02.967240+0000","last_fullsized":"2024-09-18T14:24:02.596829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:45.060182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":38875,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":498494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705885,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":180224,"data_stored":58159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.376},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301684,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070156,"statfs":{"total":96636764160,"available":96327839744,"internally_reserved":0,"allocated":176128,"data_stored":57581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:24.892 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:25.190 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:25.194 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:25.209 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-09-18T14:24:24.193445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":38875,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603372,"kb_used_data":348,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140308,"statfs":{"total":193273528320,"available":192655675392,"internally_reserved":0,"allocated":356352,"data_stored":115740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4616,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.298944+0000","last_change":"2024-09-18T14:24:04.300760+0000","last_active":"2024-09-18T14:24:04.298944+0000","last_peered":"2024-09-18T14:24:04.298944+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:04.294785+0000","last_became_peered":"2024-09-18T14:24:04.294785+0000","last_unstale":"2024-09-18T14:24:04.298944+0000","last_undegraded":"2024-09-18T14:24:03.746756+0000","last_fullsized":"2024-09-18T14:24:03.597698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:11:51.954241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4589,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.196021+0000","last_change":"2024-09-18T14:24:03.196021+0000","last_active":"2024-09-18T14:24:03.196021+0000","last_peered":"2024-09-18T14:24:03.196021+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:03.185799+0000","last_became_peered":"2024-09-18T14:24:03.185799+0000","last_unstale":"2024-09-18T14:24:03.196021+0000","last_undegraded":"2024-09-18T14:24:02.968820+0000","last_fullsized":"2024-09-18T14:24:02.598535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:22:05.974865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4482,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.312093+0000","last_change":"2024-09-18T14:24:04.313567+0000","last_active":"2024-09-18T14:24:04.312093+0000","last_peered":"2024-09-18T14:24:04.312093+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:04.308184+0000","last_became_peered":"2024-09-18T14:24:04.308184+0000","last_unstale":"2024-09-18T14:24:04.312093+0000","last_undegraded":"2024-09-18T14:24:03.738484+0000","last_fullsized":"2024-09-18T14:24:03.597525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:08:54.672615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":7341,"reported_epoch":21,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:24:23.100835+0000","last_change":"2024-09-18T14:24:03.195733+0000","last_active":"2024-09-18T14:24:23.100835+0000","last_peered":"2024-09-18T14:24:23.100835+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:03.165002+0000","last_became_peered":"2024-09-18T14:24:03.165002+0000","last_unstale":"2024-09-18T14:24:23.100835+0000","last_undegraded":"2024-09-18T14:24:23.100835+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:29:38.690657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7142,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3571},{"shards":"3","objects":932}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4514,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.213972+0000","last_change":"2024-09-18T14:24:03.213972+0000","last_active":"2024-09-18T14:24:03.213972+0000","last_peered":"2024-09-18T14:24:03.213972+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:03.206328+0000","last_became_peered":"2024-09-18T14:24:03.206328+0000","last_unstale":"2024-09-18T14:24:03.213972+0000","last_undegraded":"2024-09-18T14:24:02.948624+0000","last_fullsized":"2024-09-18T14:24:02.596557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:09:07.111338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4660,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.218843+0000","last_change":"2024-09-18T14:24:03.218843+0000","last_active":"2024-09-18T14:24:03.218843+0000","last_peered":"2024-09-18T14:24:03.218843+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:03.209404+0000","last_became_peered":"2024-09-18T14:24:03.209404+0000","last_unstale":"2024-09-18T14:24:03.218843+0000","last_undegraded":"2024-09-18T14:24:02.908616+0000","last_fullsized":"2024-09-18T14:24:02.596585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:24:21.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4636,"num_objects_misplaced":4636,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4636}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4531,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.219319+0000","last_change":"2024-09-18T14:24:03.219319+0000","last_active":"2024-09-18T14:24:03.219319+0000","last_peered":"2024-09-18T14:24:03.219319+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:03.209636+0000","last_became_peered":"2024-09-18T14:24:03.209636+0000","last_unstale":"2024-09-18T14:24:03.219319+0000","last_undegraded":"2024-09-18T14:24:02.903337+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:02:48.059016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4551,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.186314+0000","last_change":"2024-09-18T14:24:03.186314+0000","last_active":"2024-09-18T14:24:03.186314+0000","last_peered":"2024-09-18T14:24:03.186314+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:03.177160+0000","last_became_peered":"2024-09-18T14:24:03.177160+0000","last_unstale":"2024-09-18T14:24:03.186314+0000","last_undegraded":"2024-09-18T14:24:02.967240+0000","last_fullsized":"2024-09-18T14:24:02.596829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:45.060182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":38875,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":498494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705885,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":180224,"data_stored":58159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.376},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301684,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070156,"statfs":{"total":96636764160,"available":96327839744,"internally_reserved":0,"allocated":176128,"data_stored":57581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:25.210 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:24:25.210 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4616, 'reported_epoch': 21, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:04.298944+0000', 'last_change': '2024-09-18T14:24:04.300760+0000', 'last_active': '2024-09-18T14:24:04.298944+0000', 'last_peered': '2024-09-18T14:24:04.298944+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:04.294785+0000', 'last_became_peered': '2024-09-18T14:24:04.294785+0000', 'last_unstale': '2024-09-18T14:24:04.298944+0000', 'last_undegraded': '2024-09-18T14:24:03.746756+0000', 'last_fullsized': '2024-09-18T14:24:03.597698+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:11:51.954241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:25.210 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:24:25.210 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4589, 'reported_epoch': 20, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:03.196021+0000', 'last_change': '2024-09-18T14:24:03.196021+0000', 'last_active': '2024-09-18T14:24:03.196021+0000', 'last_peered': '2024-09-18T14:24:03.196021+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:03.185799+0000', 'last_became_peered': '2024-09-18T14:24:03.185799+0000', 'last_unstale': '2024-09-18T14:24:03.196021+0000', 'last_undegraded': '2024-09-18T14:24:02.968820+0000', 'last_fullsized': '2024-09-18T14:24:02.598535+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T15:22:05.974865+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:24:25.210 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4482, 'reported_epoch': 21, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:04.312093+0000', 'last_change': '2024-09-18T14:24:04.313567+0000', 'last_active': '2024-09-18T14:24:04.312093+0000', 'last_peered': '2024-09-18T14:24:04.312093+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:04.308184+0000', 'last_became_peered': '2024-09-18T14:24:04.308184+0000', 'last_unstale': '2024-09-18T14:24:04.312093+0000', 'last_undegraded': '2024-09-18T14:24:03.738484+0000', 'last_fullsized': '2024-09-18T14:24:03.597525+0000', 'mapping_epoch': 20, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:08:54.672615+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 7341, 'reported_epoch': 21, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:24:23.100835+0000', 'last_change': '2024-09-18T14:24:03.195733+0000', 'last_active': '2024-09-18T14:24:23.100835+0000', 'last_peered': '2024-09-18T14:24:23.100835+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:03.165002+0000', 'last_became_peered': '2024-09-18T14:24:03.165002+0000', 'last_unstale': '2024-09-18T14:24:23.100835+0000', 'last_undegraded': '2024-09-18T14:24:23.100835+0000', 'last_fullsized': '2024-09-18T14:24:02.598544+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T20:29:38.690657+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7142, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 932, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3571}, {'shards': '3', 'objects': 932}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4514, 'reported_epoch': 20, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:03.213972+0000', 'last_change': '2024-09-18T14:24:03.213972+0000', 'last_active': '2024-09-18T14:24:03.213972+0000', 'last_peered': '2024-09-18T14:24:03.213972+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:03.206328+0000', 'last_became_peered': '2024-09-18T14:24:03.206328+0000', 'last_unstale': '2024-09-18T14:24:03.213972+0000', 'last_undegraded': '2024-09-18T14:24:02.948624+0000', 'last_fullsized': '2024-09-18T14:24:02.596557+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T20:09:07.111338+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 4660, 'reported_epoch': 20, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:03.218843+0000', 'last_change': '2024-09-18T14:24:03.218843+0000', 'last_active': '2024-09-18T14:24:03.218843+0000', 'last_peered': '2024-09-18T14:24:03.218843+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:03.209404+0000', 'last_became_peered': '2024-09-18T14:24:03.209404+0000', 'last_unstale': '2024-09-18T14:24:03.218843+0000', 'last_undegraded': '2024-09-18T14:24:02.908616+0000', 'last_fullsized': '2024-09-18T14:24:02.596585+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-20T01:24:21.129798+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4636, 'num_objects_misplaced': 4636, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 19, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4636}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:24:25.211 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4531, 'reported_epoch': 20, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:03.219319+0000', 'last_change': '2024-09-18T14:24:03.219319+0000', 'last_active': '2024-09-18T14:24:03.219319+0000', 'last_peered': '2024-09-18T14:24:03.219319+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:03.209636+0000', 'last_became_peered': '2024-09-18T14:24:03.209636+0000', 'last_unstale': '2024-09-18T14:24:03.219319+0000', 'last_undegraded': '2024-09-18T14:24:02.903337+0000', 'last_fullsized': '2024-09-18T14:24:02.598544+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T15:02:48.059016+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:24:25.212 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:24:25.212 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4551, 'reported_epoch': 20, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:03.186314+0000', 'last_change': '2024-09-18T14:24:03.186314+0000', 'last_active': '2024-09-18T14:24:03.186314+0000', 'last_peered': '2024-09-18T14:24:03.186314+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:03.177160+0000', 'last_became_peered': '2024-09-18T14:24:03.177160+0000', 'last_unstale': '2024-09-18T14:24:03.186314+0000', 'last_undegraded': '2024-09-18T14:24:02.967240+0000', 'last_fullsized': '2024-09-18T14:24:02.596829+0000', 'mapping_epoch': 19, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T20:47:45.060182+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:25.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:25.509 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:25.512 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:25.523 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":551,"stamp":"2024-09-18T14:24:24.193445+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":38875,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603372,"kb_used_data":348,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140308,"statfs":{"total":193273528320,"available":192655675392,"internally_reserved":0,"allocated":356352,"data_stored":115740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1104,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":552,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003193"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4616,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.298944+0000","last_change":"2024-09-18T14:24:04.300760+0000","last_active":"2024-09-18T14:24:04.298944+0000","last_peered":"2024-09-18T14:24:04.298944+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:04.294785+0000","last_became_peered":"2024-09-18T14:24:04.294785+0000","last_unstale":"2024-09-18T14:24:04.298944+0000","last_undegraded":"2024-09-18T14:24:03.746756+0000","last_fullsized":"2024-09-18T14:24:03.597698+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:11:51.954241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4589,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.196021+0000","last_change":"2024-09-18T14:24:03.196021+0000","last_active":"2024-09-18T14:24:03.196021+0000","last_peered":"2024-09-18T14:24:03.196021+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:03.185799+0000","last_became_peered":"2024-09-18T14:24:03.185799+0000","last_unstale":"2024-09-18T14:24:03.196021+0000","last_undegraded":"2024-09-18T14:24:02.968820+0000","last_fullsized":"2024-09-18T14:24:02.598535+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:22:05.974865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4482,"reported_epoch":21,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:04.312093+0000","last_change":"2024-09-18T14:24:04.313567+0000","last_active":"2024-09-18T14:24:04.312093+0000","last_peered":"2024-09-18T14:24:04.312093+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:04.308184+0000","last_became_peered":"2024-09-18T14:24:04.308184+0000","last_unstale":"2024-09-18T14:24:04.312093+0000","last_undegraded":"2024-09-18T14:24:03.738484+0000","last_fullsized":"2024-09-18T14:24:03.597525+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:08:54.672615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":7341,"reported_epoch":21,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:24:23.100835+0000","last_change":"2024-09-18T14:24:03.195733+0000","last_active":"2024-09-18T14:24:23.100835+0000","last_peered":"2024-09-18T14:24:23.100835+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:03.165002+0000","last_became_peered":"2024-09-18T14:24:03.165002+0000","last_unstale":"2024-09-18T14:24:23.100835+0000","last_undegraded":"2024-09-18T14:24:23.100835+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:29:38.690657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7142,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3571},{"shards":"3","objects":932}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4514,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.213972+0000","last_change":"2024-09-18T14:24:03.213972+0000","last_active":"2024-09-18T14:24:03.213972+0000","last_peered":"2024-09-18T14:24:03.213972+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:03.206328+0000","last_became_peered":"2024-09-18T14:24:03.206328+0000","last_unstale":"2024-09-18T14:24:03.213972+0000","last_undegraded":"2024-09-18T14:24:02.948624+0000","last_fullsized":"2024-09-18T14:24:02.596557+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:09:07.111338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4660,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.218843+0000","last_change":"2024-09-18T14:24:03.218843+0000","last_active":"2024-09-18T14:24:03.218843+0000","last_peered":"2024-09-18T14:24:03.218843+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:03.209404+0000","last_became_peered":"2024-09-18T14:24:03.209404+0000","last_unstale":"2024-09-18T14:24:03.218843+0000","last_undegraded":"2024-09-18T14:24:02.908616+0000","last_fullsized":"2024-09-18T14:24:02.596585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:24:21.129798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4636,"num_objects_misplaced":4636,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4636}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4531,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.219319+0000","last_change":"2024-09-18T14:24:03.219319+0000","last_active":"2024-09-18T14:24:03.219319+0000","last_peered":"2024-09-18T14:24:03.219319+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:03.209636+0000","last_became_peered":"2024-09-18T14:24:03.209636+0000","last_unstale":"2024-09-18T14:24:03.219319+0000","last_undegraded":"2024-09-18T14:24:02.903337+0000","last_fullsized":"2024-09-18T14:24:02.598544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:02:48.059016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4551,"reported_epoch":20,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:03.186314+0000","last_change":"2024-09-18T14:24:03.186314+0000","last_active":"2024-09-18T14:24:03.186314+0000","last_peered":"2024-09-18T14:24:03.186314+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:03.177160+0000","last_became_peered":"2024-09-18T14:24:03.177160+0000","last_unstale":"2024-09-18T14:24:03.186314+0000","last_undegraded":"2024-09-18T14:24:02.967240+0000","last_fullsized":"2024-09-18T14:24:02.596829+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:45.060182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31733,"num_objects_misplaced":38875,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":498494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705885,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":180224,"data_stored":58159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.376},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705884,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301684,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070156,"statfs":{"total":96636764160,"available":96327839744,"internally_reserved":0,"allocated":176128,"data_stored":57581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.78800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:25.524 INFO:tasks.osd_recovery:stopping 2,3 2024-09-18T14:24:25.524 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T14:24:25.524 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:24:25.622 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T14:24:25.622 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T14:24:25.622 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:24:30.877 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-18T14:24:31.723 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T14:24:31.724 INFO:tasks.osd_recovery:... 2024-09-18T14:24:31.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 2 3 2024-09-18T14:24:32.028 INFO:tasks.ceph.mon.a.smithi016.stderr:2024-09-18T14:24:32.022+0000 7f755a98d700 -1 mon.a@0(leader).osd e21 definitely_dead 0 2024-09-18T14:24:32.744 INFO:teuthology.orchestra.run.smithi016.stderr:marked down osd.2. marked down osd.3. 2024-09-18T14:24:32.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-18T14:24:32.756 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-18T14:24:32.897 INFO:teuthology.orchestra.run.smithi016.stdout:38654705886 2024-09-18T14:24:32.898 INFO:teuthology.orchestra.run.smithi016.stdout:38654705888 2024-09-18T14:24:32.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:24:32.899 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:24:33.180 INFO:teuthology.orchestra.run.smithi016.stdout:38654705885 2024-09-18T14:24:33.183 INFO:teuthology.orchestra.run.smithi016.stdout:38654705887 2024-09-18T14:24:33.195 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705886 got 38654705885 for osd.0 2024-09-18T14:24:33.198 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705888 got 38654705887 for osd.1 2024-09-18T14:24:34.197 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:24:34.198 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:24:34.500 INFO:teuthology.orchestra.run.smithi016.stdout:38654705885 2024-09-18T14:24:34.513 INFO:teuthology.orchestra.run.smithi016.stdout:38654705887 2024-09-18T14:24:34.515 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705886 got 38654705885 for osd.0 2024-09-18T14:24:34.526 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705888 got 38654705887 for osd.1 2024-09-18T14:24:35.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-18T14:24:35.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-18T14:24:35.814 INFO:teuthology.orchestra.run.smithi016.stdout:38654705887 2024-09-18T14:24:35.820 INFO:teuthology.orchestra.run.smithi016.stdout:38654705888 2024-09-18T14:24:35.828 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705886 got 38654705887 for osd.0 2024-09-18T14:24:35.828 DEBUG:teuthology.parallel:result is None 2024-09-18T14:24:35.838 INFO:tasks.osd_recovery.ceph_manager:need seq 38654705888 got 38654705888 for osd.1 2024-09-18T14:24:35.838 DEBUG:teuthology.parallel:result is None 2024-09-18T14:24:35.838 INFO:tasks.osd_recovery.ceph_manager:waiting for peering to complete or become blocked 2024-09-18T14:24:35.838 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:36.122 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:36.125 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:36.136 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":558,"stamp":"2024-09-18T14:24:34.196364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603384,"kb_used_data":356,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140296,"statfs":{"total":193273528320,"available":192655663104,"internally_reserved":0,"allocated":364544,"data_stored":119098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":-31733,"num_objects_misplaced":-38875,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002434"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4614,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.738723+0000","last_change":"2024-09-18T14:24:32.738749+0000","last_active":"2024-09-18T14:24:01.590859+0000","last_peered":"2024-09-18T14:23:57.130523+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:24:32.738723+0000","last_undegraded":"2024-09-18T14:24:32.738723+0000","last_fullsized":"2024-09-18T14:24:32.738723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:43.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":4593,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4586,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735623+0000","last_change":"2024-09-18T14:24:32.735651+0000","last_active":"2024-09-18T14:24:00.775531+0000","last_peered":"2024-09-18T14:23:58.782621+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:24:32.735623+0000","last_undegraded":"2024-09-18T14:24:32.735623+0000","last_fullsized":"2024-09-18T14:24:32.735623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:13:55.455777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":4548,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4480,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735058+0000","last_change":"2024-09-18T14:24:32.735088+0000","last_active":"2024-09-18T14:24:02.597786+0000","last_peered":"2024-09-18T14:24:00.766786+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:24:32.735058+0000","last_undegraded":"2024-09-18T14:24:32.735058+0000","last_fullsized":"2024-09-18T14:24:32.735058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:03:52.843965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":4453,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4541,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735894+0000","last_change":"2024-09-18T14:24:32.735932+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:24:32.735894+0000","last_undegraded":"2024-09-18T14:24:32.735894+0000","last_fullsized":"2024-09-18T14:24:32.735894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:09:53.706550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":3442,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4511,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.739096+0000","last_change":"2024-09-18T14:24:32.739123+0000","last_active":"2024-09-18T14:24:01.590415+0000","last_peered":"2024-09-18T14:23:58.621906+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:24:32.739096+0000","last_undegraded":"2024-09-18T14:24:32.739096+0000","last_fullsized":"2024-09-18T14:24:32.739096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:08:56.080134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":4482,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4657,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.734763+0000","last_change":"2024-09-18T14:24:32.734794+0000","last_active":"2024-09-18T14:24:01.589859+0000","last_peered":"2024-09-18T14:23:59.155252+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:24:32.734763+0000","last_undegraded":"2024-09-18T14:24:32.734763+0000","last_fullsized":"2024-09-18T14:24:32.734763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:07:48.304650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":4636,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4528,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.733884+0000","last_change":"2024-09-18T14:24:32.733922+0000","last_active":"2024-09-18T14:24:00.776916+0000","last_peered":"2024-09-18T14:23:58.513183+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:24:32.733884+0000","last_undegraded":"2024-09-18T14:24:32.733884+0000","last_fullsized":"2024-09-18T14:24:32.733884+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:20.291190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":4490,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4548,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.737280+0000","last_change":"2024-09-18T14:24:32.737308+0000","last_active":"2024-09-18T14:24:01.589698+0000","last_peered":"2024-09-18T14:23:59.311146+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:24:32.737280+0000","last_undegraded":"2024-09-18T14:24:32.737280+0000","last_fullsized":"2024-09-18T14:24:32.737280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:35.644667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":4531,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":498494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705888,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":180224,"data_stored":58159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301696,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070144,"statfs":{"total":96636764160,"available":96327827456,"internally_reserved":0,"allocated":184320,"data_stored":60939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.69799999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:36.137 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:36.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:36.434 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:36.445 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":559,"stamp":"2024-09-18T14:24:36.196939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603384,"kb_used_data":356,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140296,"statfs":{"total":193273528320,"available":192655663104,"internally_reserved":0,"allocated":364544,"data_stored":119098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":-31733,"num_objects_misplaced":-38875,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002400"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4614,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.738723+0000","last_change":"2024-09-18T14:24:32.738749+0000","last_active":"2024-09-18T14:24:01.590859+0000","last_peered":"2024-09-18T14:23:57.130523+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:24:32.738723+0000","last_undegraded":"2024-09-18T14:24:32.738723+0000","last_fullsized":"2024-09-18T14:24:32.738723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:43.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":4593,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4586,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735623+0000","last_change":"2024-09-18T14:24:32.735651+0000","last_active":"2024-09-18T14:24:00.775531+0000","last_peered":"2024-09-18T14:23:58.782621+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:24:32.735623+0000","last_undegraded":"2024-09-18T14:24:32.735623+0000","last_fullsized":"2024-09-18T14:24:32.735623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:13:55.455777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":4548,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4480,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735058+0000","last_change":"2024-09-18T14:24:32.735088+0000","last_active":"2024-09-18T14:24:02.597786+0000","last_peered":"2024-09-18T14:24:00.766786+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:24:32.735058+0000","last_undegraded":"2024-09-18T14:24:32.735058+0000","last_fullsized":"2024-09-18T14:24:32.735058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:03:52.843965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":4453,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4541,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735894+0000","last_change":"2024-09-18T14:24:32.735932+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:24:32.735894+0000","last_undegraded":"2024-09-18T14:24:32.735894+0000","last_fullsized":"2024-09-18T14:24:32.735894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:09:53.706550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":3442,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4511,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.739096+0000","last_change":"2024-09-18T14:24:32.739123+0000","last_active":"2024-09-18T14:24:01.590415+0000","last_peered":"2024-09-18T14:23:58.621906+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:24:32.739096+0000","last_undegraded":"2024-09-18T14:24:32.739096+0000","last_fullsized":"2024-09-18T14:24:32.739096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:08:56.080134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":4482,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4657,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.734763+0000","last_change":"2024-09-18T14:24:32.734794+0000","last_active":"2024-09-18T14:24:01.589859+0000","last_peered":"2024-09-18T14:23:59.155252+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:24:32.734763+0000","last_undegraded":"2024-09-18T14:24:32.734763+0000","last_fullsized":"2024-09-18T14:24:32.734763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:07:48.304650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":4636,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4528,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.733884+0000","last_change":"2024-09-18T14:24:32.733922+0000","last_active":"2024-09-18T14:24:00.776916+0000","last_peered":"2024-09-18T14:23:58.513183+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:24:32.733884+0000","last_undegraded":"2024-09-18T14:24:32.733884+0000","last_fullsized":"2024-09-18T14:24:32.733884+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:20.291190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":4490,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4548,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.737280+0000","last_change":"2024-09-18T14:24:32.737308+0000","last_active":"2024-09-18T14:24:01.589698+0000","last_peered":"2024-09-18T14:23:59.311146+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:24:32.737280+0000","last_undegraded":"2024-09-18T14:24:32.737280+0000","last_fullsized":"2024-09-18T14:24:32.737280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:35.644667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":4531,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":498494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705888,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":180224,"data_stored":58159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301696,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070144,"statfs":{"total":96636764160,"available":96327827456,"internally_reserved":0,"allocated":184320,"data_stored":60939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.69799999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:36.446 INFO:tasks.osd_recovery.ceph_manager:active or down! 2024-09-18T14:24:36.446 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:36.481 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-18T14:24:36.740 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:36.741 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:36.752 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":559,"stamp":"2024-09-18T14:24:36.196939+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603384,"kb_used_data":356,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140296,"statfs":{"total":193273528320,"available":192655663104,"internally_reserved":0,"allocated":364544,"data_stored":119098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":-31733,"num_objects_misplaced":-38875,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-932,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002400"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4614,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.738723+0000","last_change":"2024-09-18T14:24:32.738749+0000","last_active":"2024-09-18T14:24:01.590859+0000","last_peered":"2024-09-18T14:23:57.130523+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:06:51.050589+0000","last_became_peered":"2024-09-18T14:06:51.050589+0000","last_unstale":"2024-09-18T14:24:32.738723+0000","last_undegraded":"2024-09-18T14:24:32.738723+0000","last_fullsized":"2024-09-18T14:24:32.738723+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T00:20:43.327913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":4593,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4586,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735623+0000","last_change":"2024-09-18T14:24:32.735651+0000","last_active":"2024-09-18T14:24:00.775531+0000","last_peered":"2024-09-18T14:23:58.782621+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:24:32.735623+0000","last_undegraded":"2024-09-18T14:24:32.735623+0000","last_fullsized":"2024-09-18T14:24:32.735623+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:13:55.455777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":4548,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4480,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735058+0000","last_change":"2024-09-18T14:24:32.735088+0000","last_active":"2024-09-18T14:24:02.597786+0000","last_peered":"2024-09-18T14:24:00.766786+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:06:40.013299+0000","last_became_peered":"2024-09-18T14:06:40.013299+0000","last_unstale":"2024-09-18T14:24:32.735058+0000","last_undegraded":"2024-09-18T14:24:32.735058+0000","last_fullsized":"2024-09-18T14:24:32.735058+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:03:52.843965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":4453,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4541,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.735894+0000","last_change":"2024-09-18T14:24:32.735932+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:24:32.735894+0000","last_undegraded":"2024-09-18T14:24:32.735894+0000","last_fullsized":"2024-09-18T14:24:32.735894+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:09:53.706550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":3442,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4511,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.739096+0000","last_change":"2024-09-18T14:24:32.739123+0000","last_active":"2024-09-18T14:24:01.590415+0000","last_peered":"2024-09-18T14:23:58.621906+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:06:51.050888+0000","last_became_peered":"2024-09-18T14:06:51.050888+0000","last_unstale":"2024-09-18T14:24:32.739096+0000","last_undegraded":"2024-09-18T14:24:32.739096+0000","last_fullsized":"2024-09-18T14:24:32.739096+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:08:56.080134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":4482,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4657,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.734763+0000","last_change":"2024-09-18T14:24:32.734794+0000","last_active":"2024-09-18T14:24:01.589859+0000","last_peered":"2024-09-18T14:23:59.155252+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:06:51.053662+0000","last_became_peered":"2024-09-18T14:06:51.053662+0000","last_unstale":"2024-09-18T14:24:32.734763+0000","last_undegraded":"2024-09-18T14:24:32.734763+0000","last_fullsized":"2024-09-18T14:24:32.734763+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:07:48.304650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":4636,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4528,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.733884+0000","last_change":"2024-09-18T14:24:32.733922+0000","last_active":"2024-09-18T14:24:00.776916+0000","last_peered":"2024-09-18T14:23:58.513183+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:24:32.733884+0000","last_undegraded":"2024-09-18T14:24:32.733884+0000","last_fullsized":"2024-09-18T14:24:32.733884+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:59:20.291190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":4490,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4548,"reported_epoch":22,"state":"down","last_fresh":"2024-09-18T14:24:32.737280+0000","last_change":"2024-09-18T14:24:32.737308+0000","last_active":"2024-09-18T14:24:01.589698+0000","last_peered":"2024-09-18T14:23:59.311146+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:06:51.050354+0000","last_became_peered":"2024-09-18T14:06:51.050354+0000","last_unstale":"2024-09-18T14:24:32.737280+0000","last_undegraded":"2024-09-18T14:24:32.737280+0000","last_fullsized":"2024-09-18T14:24:32.737280+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:19:35.644667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":4531,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":35175,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":498494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":38654705883,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705888,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301688,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070152,"statfs":{"total":96636764160,"available":96327835648,"internally_reserved":0,"allocated":180224,"data_stored":58159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.56200000000000006}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705887,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301696,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070144,"statfs":{"total":96636764160,"available":96327827456,"internally_reserved":0,"allocated":184320,"data_stored":60939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.65200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.69799999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:36.753 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-18T14:24:36.753 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-18T14:24:36.755 INFO:tasks.ceph.osd.2:Started 2024-09-18T14:24:36.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:no valid command found; 10 closest matches: 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:0 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:1 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:2 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:abort 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:assert 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:config diff 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:config diff get 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:config get 2024-09-18T14:24:36.846 INFO:teuthology.orchestra.run.smithi016.stderr:config help [] 2024-09-18T14:24:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:config set ... 2024-09-18T14:24:36.847 INFO:teuthology.orchestra.run.smithi016.stderr:admin_socket: invalid command 2024-09-18T14:24:36.848 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T14:24:36.848 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-18T14:24:37.361 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:24:37.354+0000 7f630ef60540 -1 Falling back to public interface 2024-09-18T14:24:40.671 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:24:40.666+0000 7f630ef60540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T14:24:41.850 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:no valid command found; 10 closest matches: 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:0 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:1 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:2 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:abort 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:assert 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator dump block 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator fragmentation block 2024-09-18T14:24:41.945 INFO:teuthology.orchestra.run.smithi016.stderr:bluestore allocator score block 2024-09-18T14:24:41.946 INFO:teuthology.orchestra.run.smithi016.stderr:config diff 2024-09-18T14:24:41.946 INFO:teuthology.orchestra.run.smithi016.stderr:config diff get 2024-09-18T14:24:41.946 INFO:teuthology.orchestra.run.smithi016.stderr:admin_socket: invalid command 2024-09-18T14:24:41.947 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T14:24:41.947 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-2, ['dump_ops_in_flight'] 2024-09-18T14:24:42.940 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:24:42.934+0000 7f630ef60540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T14:24:45.012 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:24:45.006+0000 7f630ef60540 -1 osd.2 21 log_to_monitors true 2024-09-18T14:24:45.282 INFO:tasks.ceph.osd.2.smithi016.stderr:2024-09-18T14:24:45.274+0000 7f63054a7700 -1 osd.2 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:24:46.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-18T14:24:47.059 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:24:47.060 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-18T14:24:47.060 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-18T14:24:47.060 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:24:47.069 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-18T14:24:47.069 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-18T14:24:47.071 INFO:tasks.ceph.osd.3:Started 2024-09-18T14:24:47.072 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T14:24:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:no valid command found; 10 closest matches: 2024-09-18T14:24:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:0 2024-09-18T14:24:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:1 2024-09-18T14:24:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:2 2024-09-18T14:24:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:abort 2024-09-18T14:24:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:assert 2024-09-18T14:24:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:config diff 2024-09-18T14:24:47.167 INFO:teuthology.orchestra.run.smithi016.stderr:config diff get 2024-09-18T14:24:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:config get 2024-09-18T14:24:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:config help [] 2024-09-18T14:24:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:config set ... 2024-09-18T14:24:47.168 INFO:teuthology.orchestra.run.smithi016.stderr:admin_socket: invalid command 2024-09-18T14:24:47.169 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T14:24:47.169 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-18T14:24:47.463 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:24:47.458+0000 7f78d7449540 -1 Falling back to public interface 2024-09-18T14:24:50.947 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:24:50.942+0000 7f78d7449540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T14:24:52.171 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T14:24:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:no valid command found; 10 closest matches: 2024-09-18T14:24:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:0 2024-09-18T14:24:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:1 2024-09-18T14:24:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:2 2024-09-18T14:24:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:abort 2024-09-18T14:24:52.281 INFO:teuthology.orchestra.run.smithi016.stderr:assert 2024-09-18T14:24:52.282 INFO:teuthology.orchestra.run.smithi016.stderr:config diff 2024-09-18T14:24:52.282 INFO:teuthology.orchestra.run.smithi016.stderr:config diff get 2024-09-18T14:24:52.282 INFO:teuthology.orchestra.run.smithi016.stderr:config get 2024-09-18T14:24:52.282 INFO:teuthology.orchestra.run.smithi016.stderr:config help [] 2024-09-18T14:24:52.282 INFO:teuthology.orchestra.run.smithi016.stderr:config set ... 2024-09-18T14:24:52.282 INFO:teuthology.orchestra.run.smithi016.stderr:admin_socket: invalid command 2024-09-18T14:24:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-18T14:24:52.283 INFO:tasks.osd_recovery.ceph_manager:waiting on admin_socket for osd-3, ['dump_ops_in_flight'] 2024-09-18T14:24:53.258 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:24:53.250+0000 7f78d7449540 -1 bluestore::NCB::__restore_allocator::No Valid allocation info on disk (empty file) 2024-09-18T14:24:55.514 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:24:55.506+0000 7f78d7449540 -1 osd.3 21 log_to_monitors true 2024-09-18T14:24:56.408 INFO:tasks.ceph.osd.3.smithi016.stderr:2024-09-18T14:24:56.402+0000 7f78cf994700 -1 osd.3 21 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-18T14:24:57.285 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-18T14:24:57.427 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:24:57.427 INFO:teuthology.orchestra.run.smithi016.stdout: "ops": [], 2024-09-18T14:24:57.427 INFO:teuthology.orchestra.run.smithi016.stdout: "num_ops": 0 2024-09-18T14:24:57.428 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:24:57.444 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-18T14:24:57.751 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 29 2024-09-18T14:24:57.751 INFO:teuthology.orchestra.run.smithi016.stdout:fsid 946c9667-3cd7-48a9-8c9c-fd6b4af766c9 2024-09-18T14:24:57.751 INFO:teuthology.orchestra.run.smithi016.stdout:created 2024-09-18T14:06:29.125568+0000 2024-09-18T14:24:57.751 INFO:teuthology.orchestra.run.smithi016.stdout:modified 2024-09-18T14:24:57.391001+0000 2024-09-18T14:24:57.751 INFO:teuthology.orchestra.run.smithi016.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-18T14:24:57.751 INFO:teuthology.orchestra.run.smithi016.stdout:crush_version 6 2024-09-18T14:24:57.751 INFO:teuthology.orchestra.run.smithi016.stdout:full_ratio 0.95 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:backfillfull_ratio 0.9 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:nearfull_ratio 0.85 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:require_min_compat_client luminous 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:min_compat_client jewel 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:require_osd_release quincy 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:stretch_mode_enabled false 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 14 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:max_osd 4 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd.0 up in weight 1 up_from 9 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/22294,v1:172.21.15.16:6809/22294] [v2:172.21.15.16:6810/22294,v1:172.21.15.16:6811/22294] exists,up 08356eed-f55f-427f-bce6-f522a976a2c9 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd.1 up in weight 1 up_from 9 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6824/22293,v1:172.21.15.16:6825/22293] [v2:172.21.15.16:6826/22293,v1:172.21.15.16:6827/22293] exists,up 851cad7b-f44e-48a3-9d7a-2498617c734e 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd.2 up out weight 0 up_from 25 up_thru 26 down_at 22 last_clean_interval [9,21) [v2:172.21.15.16:6800/53651,v1:172.21.15.16:6801/53651] [v2:172.21.15.16:6802/53651,v1:172.21.15.16:6803/53651] exists,up 8dc73a5a-05ad-471a-8d5b-442d5605ab45 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:osd.3 up out weight 0 up_from 29 up_thru 19 down_at 22 last_clean_interval [9,21) [v2:172.21.15.16:6816/54496,v1:172.21.15.16:6817/54496] [v2:172.21.15.16:6818/54496,v1:172.21.15.16:6819/54496] exists,up 536c2bb9-0d34-4dc1-b8b2-f2df92543320 2024-09-18T14:24:57.752 INFO:teuthology.orchestra.run.smithi016.stdout:pg_temp 1.0 [2] 2024-09-18T14:24:57.753 INFO:teuthology.orchestra.run.smithi016.stdout:pg_temp 1.2 [2] 2024-09-18T14:24:57.753 INFO:teuthology.orchestra.run.smithi016.stdout:pg_temp 1.3 [2] 2024-09-18T14:24:57.753 INFO:teuthology.orchestra.run.smithi016.stdout:pg_temp 1.5 [2] 2024-09-18T14:24:57.753 INFO:teuthology.orchestra.run.smithi016.stdout:pg_temp 1.7 [2] 2024-09-18T14:24:57.766 INFO:tasks.osd_recovery.ceph_manager:['osd.0 up in weight 1 up_from 9 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6808/22294,v1:172.21.15.16:6809/22294] [v2:172.21.15.16:6810/22294,v1:172.21.15.16:6811/22294] exists,up 08356eed-f55f-427f-bce6-f522a976a2c9', 'osd.1 up in weight 1 up_from 9 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:172.21.15.16:6824/22293,v1:172.21.15.16:6825/22293] [v2:172.21.15.16:6826/22293,v1:172.21.15.16:6827/22293] exists,up 851cad7b-f44e-48a3-9d7a-2498617c734e', 'osd.2 up out weight 0 up_from 25 up_thru 26 down_at 22 last_clean_interval [9,21) [v2:172.21.15.16:6800/53651,v1:172.21.15.16:6801/53651] [v2:172.21.15.16:6802/53651,v1:172.21.15.16:6803/53651] exists,up 8dc73a5a-05ad-471a-8d5b-442d5605ab45', 'osd.3 up out weight 0 up_from 29 up_thru 19 down_at 22 last_clean_interval [9,21) [v2:172.21.15.16:6816/54496,v1:172.21.15.16:6817/54496] [v2:172.21.15.16:6818/54496,v1:172.21.15.16:6819/54496] exists,up 536c2bb9-0d34-4dc1-b8b2-f2df92543320'] 2024-09-18T14:24:57.766 INFO:tasks.osd_recovery:all are up! 2024-09-18T14:24:57.766 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 debug kick_recovery_wq 0 2024-09-18T14:24:57.910 INFO:teuthology.orchestra.run.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:24:57.919 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 debug kick_recovery_wq 0 2024-09-18T14:24:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:24:58.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 debug kick_recovery_wq 0 2024-09-18T14:24:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:24:58.249 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 debug kick_recovery_wq 0 2024-09-18T14:24:58.388 INFO:teuthology.orchestra.run.smithi016.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-09-18T14:24:58.401 INFO:tasks.osd_recovery.ceph_manager:waiting for clean 2024-09-18T14:24:58.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:58.724 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:58.727 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:58.741 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2024-09-18T14:24:58.405596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":12480,"num_objects_degraded":22274,"num_objects_misplaced":22274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603496,"kb_used_data":468,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140184,"statfs":{"total":193273528320,"available":192655548416,"internally_reserved":0,"allocated":479232,"data_stored":165138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":-270,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.204221"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4626,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.472801+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:48.472801+0000","last_peered":"2024-09-18T14:24:48.472801+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:48.472801+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:50.235761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4594,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401154+0000","last_change":"2024-09-18T14:24:57.404523+0000","last_active":"2024-09-18T14:24:00.775531+0000","last_peered":"2024-09-18T14:23:58.782621+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:24:56.401154+0000","last_undegraded":"2024-09-18T14:24:56.401154+0000","last_fullsized":"2024-09-18T14:24:56.401154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:23.762612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":4548,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4492,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.468928+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:48.468928+0000","last_peered":"2024-09-18T14:24:48.468928+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:48.468928+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:20:19.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4549,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401161+0000","last_change":"2024-09-18T14:24:57.404117+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:24:56.401161+0000","last_undegraded":"2024-09-18T14:24:56.401161+0000","last_fullsized":"2024-09-18T14:24:56.401161+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:11:13.155113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":3442,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4524,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.418306+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:48.418306+0000","last_peered":"2024-09-18T14:24:48.418306+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:48.418306+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:46.381668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":5934,"reported_epoch":27,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:56.290767+0000","last_change":"2024-09-18T14:24:48.421886+0000","last_active":"2024-09-18T14:24:56.290767+0000","last_peered":"2024-09-18T14:24:56.290767+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:24:56.290767+0000","last_undegraded":"2024-09-18T14:24:48.404976+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:56:29.152265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4215,"num_objects_misplaced":4215,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":421}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4536,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401174+0000","last_change":"2024-09-18T14:24:57.404885+0000","last_active":"2024-09-18T14:24:00.776916+0000","last_peered":"2024-09-18T14:23:58.513183+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:24:56.401174+0000","last_undegraded":"2024-09-18T14:24:56.401174+0000","last_fullsized":"2024-09-18T14:24:56.401174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:23:47.400248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":4490,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4561,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.466349+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:48.466349+0000","last_peered":"2024-09-18T14:24:48.466349+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:48.466349+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:53.455357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":12480,"num_objects_degraded":22274,"num_objects_misplaced":22274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051586,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182403,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705893,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":241664,"data_stored":82858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.254},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":237568,"data_stored":82280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.313}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:58.742 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:59.063 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:59.067 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:59.081 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":577,"stamp":"2024-09-18T14:24:58.405596+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":12480,"num_objects_degraded":22274,"num_objects_misplaced":22274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603496,"kb_used_data":468,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140184,"statfs":{"total":193273528320,"available":192655548416,"internally_reserved":0,"allocated":479232,"data_stored":165138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":-270,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.204221"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4626,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.472801+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:48.472801+0000","last_peered":"2024-09-18T14:24:48.472801+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:48.472801+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:50.235761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4594,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401154+0000","last_change":"2024-09-18T14:24:57.404523+0000","last_active":"2024-09-18T14:24:00.775531+0000","last_peered":"2024-09-18T14:23:58.782621+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:24:56.401154+0000","last_undegraded":"2024-09-18T14:24:56.401154+0000","last_fullsized":"2024-09-18T14:24:56.401154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:23.762612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":4548,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4492,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.468928+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:48.468928+0000","last_peered":"2024-09-18T14:24:48.468928+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:48.468928+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:20:19.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4549,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401161+0000","last_change":"2024-09-18T14:24:57.404117+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:24:56.401161+0000","last_undegraded":"2024-09-18T14:24:56.401161+0000","last_fullsized":"2024-09-18T14:24:56.401161+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:11:13.155113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":3442,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4524,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.418306+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:48.418306+0000","last_peered":"2024-09-18T14:24:48.418306+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:48.418306+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:46.381668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":5934,"reported_epoch":27,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:56.290767+0000","last_change":"2024-09-18T14:24:48.421886+0000","last_active":"2024-09-18T14:24:56.290767+0000","last_peered":"2024-09-18T14:24:56.290767+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:24:56.290767+0000","last_undegraded":"2024-09-18T14:24:48.404976+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:56:29.152265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4215,"num_objects_misplaced":4215,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":421}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4536,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401174+0000","last_change":"2024-09-18T14:24:57.404885+0000","last_active":"2024-09-18T14:24:00.776916+0000","last_peered":"2024-09-18T14:23:58.513183+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:24:56.401174+0000","last_undegraded":"2024-09-18T14:24:56.401174+0000","last_fullsized":"2024-09-18T14:24:56.401174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:23:47.400248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":4490,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4561,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.466349+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:48.466349+0000","last_peered":"2024-09-18T14:24:48.466349+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:48.466349+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:53.455357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":12480,"num_objects_degraded":22274,"num_objects_misplaced":22274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051586,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182403,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705893,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":241664,"data_stored":82858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.254},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":237568,"data_stored":82280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.313}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:59.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:24:59.432 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:59.436 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:24:59.450 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":578,"stamp":"2024-09-18T14:24:59.407647+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":12480,"num_objects_degraded":22274,"num_objects_misplaced":22274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603496,"kb_used_data":468,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140184,"statfs":{"total":193273528320,"available":192655548416,"internally_reserved":0,"allocated":479232,"data_stored":165138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":-270,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.205913"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4626,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.472801+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:48.472801+0000","last_peered":"2024-09-18T14:24:48.472801+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:48.472801+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:50.235761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4594,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401154+0000","last_change":"2024-09-18T14:24:57.404523+0000","last_active":"2024-09-18T14:24:00.775531+0000","last_peered":"2024-09-18T14:23:58.782621+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:24:56.401154+0000","last_undegraded":"2024-09-18T14:24:56.401154+0000","last_fullsized":"2024-09-18T14:24:56.401154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:23.762612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":4548,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4492,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.468928+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:48.468928+0000","last_peered":"2024-09-18T14:24:48.468928+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:48.468928+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:20:19.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4549,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401161+0000","last_change":"2024-09-18T14:24:57.404117+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:24:56.401161+0000","last_undegraded":"2024-09-18T14:24:56.401161+0000","last_fullsized":"2024-09-18T14:24:56.401161+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:11:13.155113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":3442,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4524,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.418306+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:48.418306+0000","last_peered":"2024-09-18T14:24:48.418306+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:48.418306+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:46.381668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":5934,"reported_epoch":27,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:56.290767+0000","last_change":"2024-09-18T14:24:48.421886+0000","last_active":"2024-09-18T14:24:56.290767+0000","last_peered":"2024-09-18T14:24:56.290767+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:24:56.290767+0000","last_undegraded":"2024-09-18T14:24:48.404976+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:56:29.152265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4215,"num_objects_misplaced":4215,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":421}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4536,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401174+0000","last_change":"2024-09-18T14:24:57.404885+0000","last_active":"2024-09-18T14:24:00.776916+0000","last_peered":"2024-09-18T14:23:58.513183+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:24:56.401174+0000","last_undegraded":"2024-09-18T14:24:56.401174+0000","last_fullsized":"2024-09-18T14:24:56.401174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:23:47.400248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":4490,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4561,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.466349+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:48.466349+0000","last_peered":"2024-09-18T14:24:48.466349+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:48.466349+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:53.455357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":12480,"num_objects_degraded":22274,"num_objects_misplaced":22274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051586,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182403,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705893,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":241664,"data_stored":82858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.254},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":237568,"data_stored":82280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.313}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:24:59.451 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:24:59.451 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4626, 'reported_epoch': 27, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:48.472801+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:48.472801+0000', 'last_peered': '2024-09-18T14:24:48.472801+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:48.472801+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:14:50.235761+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:59.451 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:24:59.451 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4594, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-18T14:24:56.401154+0000', 'last_change': '2024-09-18T14:24:57.404523+0000', 'last_active': '2024-09-18T14:24:00.775531+0000', 'last_peered': '2024-09-18T14:23:58.782621+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:06:51.051688+0000', 'last_became_peered': '2024-09-18T14:06:51.051688+0000', 'last_unstale': '2024-09-18T14:24:56.401154+0000', 'last_undegraded': '2024-09-18T14:24:56.401154+0000', 'last_fullsized': '2024-09-18T14:24:56.401154+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:42:23.762612+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 4548, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T14:24:59.451 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:24:59.451 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4492, 'reported_epoch': 27, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:48.468928+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:48.468928+0000', 'last_peered': '2024-09-18T14:24:48.468928+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:48.468928+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T15:20:19.812495+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:59.451 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:24:59.451 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4549, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-18T14:24:56.401161+0000', 'last_change': '2024-09-18T14:24:57.404117+0000', 'last_active': '2024-09-18T14:24:00.775289+0000', 'last_peered': '2024-09-18T14:23:59.258639+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:06:51.051512+0000', 'last_became_peered': '2024-09-18T14:06:51.051512+0000', 'last_unstale': '2024-09-18T14:24:56.401161+0000', 'last_undegraded': '2024-09-18T14:24:56.401161+0000', 'last_fullsized': '2024-09-18T14:24:56.401161+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-20T01:11:13.155113+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 3442, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T14:24:59.452 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:24:59.452 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4524, 'reported_epoch': 27, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:48.418306+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:48.418306+0000', 'last_peered': '2024-09-18T14:24:48.418306+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:48.418306+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:14:46.381668+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:59.452 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:24:59.452 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 5934, 'reported_epoch': 27, 'state': 'active+recovering+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:56.290767+0000', 'last_change': '2024-09-18T14:24:48.421886+0000', 'last_active': '2024-09-18T14:24:56.290767+0000', 'last_peered': '2024-09-18T14:24:56.290767+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:24:56.290767+0000', 'last_undegraded': '2024-09-18T14:24:48.404976+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:56:29.152265+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4215, 'num_objects_misplaced': 4215, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 422, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 421}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:59.452 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:24:59.452 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4536, 'reported_epoch': 28, 'state': 'peering', 'last_fresh': '2024-09-18T14:24:56.401174+0000', 'last_change': '2024-09-18T14:24:57.404885+0000', 'last_active': '2024-09-18T14:24:00.776916+0000', 'last_peered': '2024-09-18T14:23:58.513183+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:06:51.052357+0000', 'last_became_peered': '2024-09-18T14:06:51.052357+0000', 'last_unstale': '2024-09-18T14:24:56.401174+0000', 'last_undegraded': '2024-09-18T14:24:56.401174+0000', 'last_fullsized': '2024-09-18T14:24:56.401174+0000', 'mapping_epoch': 22, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-20T01:23:47.400248+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 4490, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [1, 2, 3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T14:24:59.452 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:24:59.452 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4561, 'reported_epoch': 27, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:48.466349+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:48.466349+0000', 'last_peered': '2024-09-18T14:24:48.466349+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:48.466349+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T16:58:53.455357+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [{'shards': '2', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:24:59.453 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:24:59.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:24:59.840 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22274/72472 objects degraded (30.735%), 5 pgs degraded","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1110,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"peering","count":3},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":617979904,"bytes_avail":192655548416,"bytes_total":193273528320,"inactive_pgs_ratio":0.375,"degraded_objects":22274,"degraded_total":72472,"degraded_ratio":0.30734628546197151,"misplaced_objects":22274,"misplaced_total":72472,"misplaced_ratio":0.30734628546197151,"recovering_objects_per_sec":52,"recovering_bytes_per_sec":2,"recovering_keys_per_sec":0,"num_objects_recovered":421,"num_bytes_recovered":19,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:24:59.855 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:24:59.856 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:00.985 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:01.003 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":579,"stamp":"2024-09-18T14:25:00.203521+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":12480,"num_objects_degraded":22274,"num_objects_misplaced":22274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603496,"kb_used_data":468,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140184,"statfs":{"total":193273528320,"available":192655548416,"internally_reserved":0,"allocated":479232,"data_stored":165138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-269,"num_objects_misplaced":-270,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":270,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001105"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4626,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.472801+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:48.472801+0000","last_peered":"2024-09-18T14:24:48.472801+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:48.472801+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:50.235761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4594,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401154+0000","last_change":"2024-09-18T14:24:57.404523+0000","last_active":"2024-09-18T14:24:00.775531+0000","last_peered":"2024-09-18T14:23:58.782621+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:06:51.051688+0000","last_became_peered":"2024-09-18T14:06:51.051688+0000","last_unstale":"2024-09-18T14:24:56.401154+0000","last_undegraded":"2024-09-18T14:24:56.401154+0000","last_fullsized":"2024-09-18T14:24:56.401154+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:23.762612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":4548,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4492,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.468928+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:48.468928+0000","last_peered":"2024-09-18T14:24:48.468928+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:48.468928+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T15:20:19.812495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4549,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401161+0000","last_change":"2024-09-18T14:24:57.404117+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:24:56.401161+0000","last_undegraded":"2024-09-18T14:24:56.401161+0000","last_fullsized":"2024-09-18T14:24:56.401161+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:11:13.155113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":3442,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4524,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.418306+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:48.418306+0000","last_peered":"2024-09-18T14:24:48.418306+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:48.418306+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:14:46.381668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":5934,"reported_epoch":27,"state":"active+recovering+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:56.290767+0000","last_change":"2024-09-18T14:24:48.421886+0000","last_active":"2024-09-18T14:24:56.290767+0000","last_peered":"2024-09-18T14:24:56.290767+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:24:56.290767+0000","last_undegraded":"2024-09-18T14:24:48.404976+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:56:29.152265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4215,"num_objects_misplaced":4215,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":422,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":421}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4536,"reported_epoch":28,"state":"peering","last_fresh":"2024-09-18T14:24:56.401174+0000","last_change":"2024-09-18T14:24:57.404885+0000","last_active":"2024-09-18T14:24:00.776916+0000","last_peered":"2024-09-18T14:23:58.513183+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:06:51.052357+0000","last_became_peered":"2024-09-18T14:06:51.052357+0000","last_unstale":"2024-09-18T14:24:56.401174+0000","last_undegraded":"2024-09-18T14:24:56.401174+0000","last_fullsized":"2024-09-18T14:24:56.401174+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:23:47.400248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":4490,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4561,"reported_epoch":27,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:48.466349+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:48.466349+0000","last_peered":"2024-09-18T14:24:48.466349+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:48.466349+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:58:53.455357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":12480,"num_objects_degraded":22274,"num_objects_misplaced":22274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":423,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051586,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182403,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705893,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":241664,"data_stored":82858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.254},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.26800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.62},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705892,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301748,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070092,"statfs":{"total":96636764160,"available":96327774208,"internally_reserved":0,"allocated":237568,"data_stored":82280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.23200000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.40200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.313}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:04.004 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:04.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:04.305 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:04.321 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2024-09-18T14:25:04.204408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":38445,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":685,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611860,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131820,"statfs":{"total":193273528320,"available":192646983680,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":16171,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.809696"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":6724,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:01.298064+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:01.298064+0000","last_peered":"2024-09-18T14:25:01.298064+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:01.298064+0000","last_undegraded":"2024-09-18T14:25:01.298064+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7906,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":683},{"shards":"2,3","objects":3953}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":38445,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":685,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051587,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182404,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305932,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065908,"statfs":{"total":96636764160,"available":96323489792,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.42099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.374},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.45700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.623},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:04.322 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:04.607 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:04.610 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:04.620 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2024-09-18T14:25:04.204408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":38445,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":685,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611860,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131820,"statfs":{"total":193273528320,"available":192646983680,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":16171,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.809696"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":6724,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:01.298064+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:01.298064+0000","last_peered":"2024-09-18T14:25:01.298064+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:01.298064+0000","last_undegraded":"2024-09-18T14:25:01.298064+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7906,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":683},{"shards":"2,3","objects":3953}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":38445,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":685,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051587,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182404,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305932,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065908,"statfs":{"total":96636764160,"available":96323489792,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.42099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.374},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.45700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.623},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:04.621 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:04.621 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:04.621 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:04.621 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 6724, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:01.298064+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:01.298064+0000', 'last_peered': '2024-09-18T14:25:01.298064+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:01.298064+0000', 'last_undegraded': '2024-09-18T14:25:01.298064+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7906, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 684, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 683}, {'shards': '2,3', 'objects': 3953}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:04.622 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:04.623 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:04.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:04.623 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:04.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:04.971 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:04.972 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1115,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626544640,"bytes_avail":192646983680,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":38445,"misplaced_total":72472,"misplaced_ratio":0.53048073738823265,"recovering_objects_per_sec":38,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:04.988 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:04.988 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:05.291 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:05.295 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:05.305 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":581,"stamp":"2024-09-18T14:25:04.204408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":38445,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":685,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611860,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131820,"statfs":{"total":193273528320,"available":192646983680,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":16171,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.809696"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":6724,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:01.298064+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:01.298064+0000","last_peered":"2024-09-18T14:25:01.298064+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:01.298064+0000","last_undegraded":"2024-09-18T14:25:01.298064+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7906,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":684,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":683},{"shards":"2,3","objects":3953}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":38445,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":685,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051587,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182404,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305932,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065908,"statfs":{"total":96636764160,"available":96323489792,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.30599999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.42099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.374},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.45700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705893,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.67500000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.623},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.54500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:08.307 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:08.643 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:08.646 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:08.656 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2024-09-18T14:25:08.205462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37871,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":15597,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.799866"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":7585,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:06.307056+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:06.307056+0000","last_peered":"2024-09-18T14:25:06.307056+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:06.307056+0000","last_undegraded":"2024-09-18T14:25:06.307056+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7332,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":970},{"shards":"2,3","objects":3666}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37871,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051588,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182405,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705895,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.377},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.498}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.54400000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.58199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:08.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:08.952 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:08.969 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2024-09-18T14:25:08.205462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37871,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":15597,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.799866"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":7585,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:06.307056+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:06.307056+0000","last_peered":"2024-09-18T14:25:06.307056+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:06.307056+0000","last_undegraded":"2024-09-18T14:25:06.307056+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7332,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":970},{"shards":"2,3","objects":3666}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37871,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051588,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182405,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705895,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.377},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.498}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.54400000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.58199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:08.970 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:08.971 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:08.971 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:08.971 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 7585, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:06.307056+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:06.307056+0000', 'last_peered': '2024-09-18T14:25:06.307056+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:06.307056+0000', 'last_undegraded': '2024-09-18T14:25:06.307056+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7332, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 971, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 970}, {'shards': '2,3', 'objects': 3666}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:08.971 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:08.971 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:08.971 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:08.971 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:08.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:09.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:09.349 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1120,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626540544,"bytes_avail":192646987776,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":37871,"misplaced_total":72472,"misplaced_ratio":0.52256043713434153,"recovering_objects_per_sec":56,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":549,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:09.363 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:09.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:09.655 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:09.659 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:09.671 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":583,"stamp":"2024-09-18T14:25:08.205462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37871,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":15597,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":549,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.799866"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":7585,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:06.307056+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:06.307056+0000","last_peered":"2024-09-18T14:25:06.307056+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:06.307056+0000","last_undegraded":"2024-09-18T14:25:06.307056+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7332,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":970},{"shards":"2,3","objects":3666}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37871,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":972,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051588,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182405,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705895,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.60299999999999998,"15min":0.57299999999999995},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":6.758,"5min":6.758,"15min":6.758},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.84699999999999998,"5min":0.65700000000000003,"15min":0.627},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":6.7859999999999996,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.377},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.498}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.54400000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.58199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:12.673 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:12.997 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:13.001 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:13.012 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2024-09-18T14:25:12.206456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":15013,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":8461,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:11.307711+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:11.307711+0000","last_peered":"2024-09-18T14:25:11.307711+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:11.307711+0000","last_undegraded":"2024-09-18T14:25:11.307711+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6748,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1262},{"shards":"2,3","objects":3374}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051588,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.753}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.70999999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.58199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:13.013 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:13.308 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:13.312 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:13.322 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2024-09-18T14:25:12.206456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":15013,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":8461,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:11.307711+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:11.307711+0000","last_peered":"2024-09-18T14:25:11.307711+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:11.307711+0000","last_undegraded":"2024-09-18T14:25:11.307711+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6748,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1262},{"shards":"2,3","objects":3374}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051588,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.753}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.70999999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.58199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:13.323 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:13.323 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:13.323 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:13.323 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:13.323 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:13.323 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:13.323 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:13.323 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:13.324 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:13.324 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:13.324 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:13.324 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 8461, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:11.307711+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:11.307711+0000', 'last_peered': '2024-09-18T14:25:11.307711+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:11.307711+0000', 'last_undegraded': '2024-09-18T14:25:11.307711+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6748, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1263, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1262}, {'shards': '2,3', 'objects': 3374}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:13.324 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:13.324 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:13.324 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:13.324 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:13.325 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:13.678 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:13.678 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1124,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626540544,"bytes_avail":192646987776,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":37287,"misplaced_total":72472,"misplaced_ratio":0.51450215255546972,"recovering_objects_per_sec":70,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":841,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:13.696 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:13.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:13.995 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:13.998 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:14.009 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":585,"stamp":"2024-09-18T14:25:12.206456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12480,"num_objects_degraded":8265,"num_objects_misplaced":15013,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002935"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":8461,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:11.307711+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:11.307711+0000","last_peered":"2024-09-18T14:25:11.307711+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:11.307711+0000","last_undegraded":"2024-09-18T14:25:11.307711+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6748,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1262},{"shards":"2,3","objects":3374}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051588,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.753}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.70999999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705894,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.39000000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.66600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.58199999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:17.010 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:17.324 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:17.329 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:17.340 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2024-09-18T14:25:16.207753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1158,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003345"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":8461,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:11.307711+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:11.307711+0000","last_peered":"2024-09-18T14:25:11.307711+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:11.307711+0000","last_undegraded":"2024-09-18T14:25:11.307711+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6748,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1262},{"shards":"2,3","objects":3374}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051589,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.753}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.70999999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.54200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.76100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:17.341 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:17.632 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:17.634 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:17.647 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":587,"stamp":"2024-09-18T14:25:16.207753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1158,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":579,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003345"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":8461,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:11.307711+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:11.307711+0000","last_peered":"2024-09-18T14:25:11.307711+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:11.307711+0000","last_undegraded":"2024-09-18T14:25:11.307711+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6748,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1263,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1262},{"shards":"2,3","objects":3374}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":37287,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1264,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051589,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182406,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.753}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.70999999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705895,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.66600000000000004,"15min":0.68600000000000005},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.79100000000000004,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.54200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.76100000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.81100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:17.648 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:17.649 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:17.649 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:17.649 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 8461, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:11.307711+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:11.307711+0000', 'last_peered': '2024-09-18T14:25:11.307711+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:11.307711+0000', 'last_undegraded': '2024-09-18T14:25:11.307711+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6748, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1263, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1262}, {'shards': '2,3', 'objects': 3374}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:17.649 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:17.649 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:17.649 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:17.649 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:17.649 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:18.008 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:18.009 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1128,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626540544,"bytes_avail":192646987776,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":37287,"misplaced_total":72472,"misplaced_ratio":0.51450215255546972,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":579,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:18.022 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:18.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:18.332 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:18.336 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:18.351 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":588,"stamp":"2024-09-18T14:25:18.208317+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36655,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1790,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":895,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003452"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":9409,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:16.322333+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:16.322333+0000","last_peered":"2024-09-18T14:25:16.322333+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:16.322333+0000","last_undegraded":"2024-09-18T14:25:16.322333+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6116,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1579,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1578},{"shards":"2,3","objects":3058}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36655,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051590,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705897,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.441},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.54300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.502},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.56699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:21.353 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:21.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:21.696 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:21.706 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2024-09-18T14:25:20.208930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36655,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1216,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003468"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":9409,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:16.322333+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:16.322333+0000","last_peered":"2024-09-18T14:25:16.322333+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:16.322333+0000","last_undegraded":"2024-09-18T14:25:16.322333+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6116,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1579,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1578},{"shards":"2,3","objects":3058}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36655,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051590,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705897,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.441},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.54300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.502},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.56699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:21.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:22.028 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:22.032 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:22.046 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":589,"stamp":"2024-09-18T14:25:20.208930+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36655,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611856,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131824,"statfs":{"total":193273528320,"available":192646987776,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1216,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003468"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":9409,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:16.322333+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:16.322333+0000","last_peered":"2024-09-18T14:25:16.322333+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:16.322333+0000","last_undegraded":"2024-09-18T14:25:16.322333+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6116,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1579,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1578},{"shards":"2,3","objects":3058}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36655,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1580,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051590,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182407,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705897,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.441},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.54300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.502},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.56699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:22.047 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:22.047 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:22.048 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 9409, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:16.322333+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:16.322333+0000', 'last_peered': '2024-09-18T14:25:16.322333+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:16.322333+0000', 'last_undegraded': '2024-09-18T14:25:16.322333+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6116, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1579, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1578}, {'shards': '2,3', 'objects': 3058}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:22.049 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:22.049 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:22.049 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:22.049 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:22.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:22.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:22.428 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1133,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626540544,"bytes_avail":192646987776,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":36655,"misplaced_total":72472,"misplaced_ratio":0.50578154321669055,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":608,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:22.446 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:22.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:22.744 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:22.748 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:22.759 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":590,"stamp":"2024-09-18T14:25:22.209341+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36001,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616016,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188127664,"statfs":{"total":193273528320,"available":192642727936,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1870,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":935,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003309"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":10390,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:21.325050+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:21.325050+0000","last_peered":"2024-09-18T14:25:21.325050+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:21.325050+0000","last_undegraded":"2024-09-18T14:25:21.325050+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5462,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1906,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1905},{"shards":"2,3","objects":2731}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36001,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051590,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182408,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.307},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.502},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.58399999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705896,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305928,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065912,"statfs":{"total":96636764160,"available":96323493888,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.54400000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.502},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.56699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:25.761 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:26.075 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:26.078 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:26.090 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":591,"stamp":"2024-09-18T14:25:24.209906+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36001,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620176,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123504,"statfs":{"total":193273528320,"available":192638468096,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1286,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003449"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":10390,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:21.325050+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:21.325050+0000","last_peered":"2024-09-18T14:25:21.325050+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:21.325050+0000","last_undegraded":"2024-09-18T14:25:21.325050+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5462,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1906,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1905},{"shards":"2,3","objects":2731}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36001,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051591,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182408,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.307},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.502},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.58399999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.748},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.69799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.85199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:26.091 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:26.396 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:26.400 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:26.409 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2024-09-18T14:25:26.210513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36001,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620176,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123504,"statfs":{"total":193273528320,"available":192638468096,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1286,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003349"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":10390,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:21.325050+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:21.325050+0000","last_peered":"2024-09-18T14:25:21.325050+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:21.325050+0000","last_undegraded":"2024-09-18T14:25:21.325050+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5462,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1906,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1905},{"shards":"2,3","objects":2731}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36001,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051591,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182408,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.307},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.502},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.58399999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.748},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.69799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.85199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:26.410 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:26.410 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:26.411 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 10390, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:21.325050+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:21.325050+0000', 'last_peered': '2024-09-18T14:25:21.325050+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:21.325050+0000', 'last_undegraded': '2024-09-18T14:25:21.325050+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5462, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1906, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1905}, {'shards': '2,3', 'objects': 2731}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:26.412 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:26.412 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:26.412 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:26.412 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:26.412 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:26.796 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:26.796 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1137,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635060224,"bytes_avail":192638468096,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":36001,"misplaced_total":72472,"misplaced_ratio":0.49675736836295398,"recovering_objects_per_sec":53,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":643,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:26.810 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:26.811 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:27.121 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:27.125 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:27.136 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":592,"stamp":"2024-09-18T14:25:26.210513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36001,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620176,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123504,"statfs":{"total":193273528320,"available":192638468096,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1286,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":643,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003349"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":10390,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:21.325050+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:21.325050+0000","last_peered":"2024-09-18T14:25:21.325050+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:21.325050+0000","last_undegraded":"2024-09-18T14:25:21.325050+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5462,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1906,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1905},{"shards":"2,3","objects":2731}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":36001,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1907,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051591,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182408,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.307},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.502},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.26600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.58399999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705897,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.748},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.69799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.85199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:30.138 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:30.443 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:30.446 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:30.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2024-09-18T14:25:30.211442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":35303,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620176,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123504,"statfs":{"total":193273528320,"available":192638468096,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1352,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003124"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":11437,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:26.335626+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:26.335626+0000","last_peered":"2024-09-18T14:25:26.335626+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:26.335626+0000","last_undegraded":"2024-09-18T14:25:26.335626+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4764,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2255,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2254},{"shards":"2,3","objects":2382}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":35303,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051592,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.434},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.67400000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.88800000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.628}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.39500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:30.458 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:30.799 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:30.803 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:30.813 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2024-09-18T14:25:30.211442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":35303,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620176,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123504,"statfs":{"total":193273528320,"available":192638468096,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1352,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003124"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":11437,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:26.335626+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:26.335626+0000","last_peered":"2024-09-18T14:25:26.335626+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:26.335626+0000","last_undegraded":"2024-09-18T14:25:26.335626+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4764,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2255,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2254},{"shards":"2,3","objects":2382}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":35303,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051592,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.434},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.67400000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.88800000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.628}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.39500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:30.814 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:30.814 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:30.814 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:30.814 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:30.814 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:30.814 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:30.814 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:30.814 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:30.815 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:30.815 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:30.815 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:30.815 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 11437, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:26.335626+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:26.335626+0000', 'last_peered': '2024-09-18T14:25:26.335626+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:26.335626+0000', 'last_undegraded': '2024-09-18T14:25:26.335626+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4764, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2255, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2254}, {'shards': '2,3', 'objects': 2382}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:30.815 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:30.815 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:30.815 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:30.815 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:30.815 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:31.184 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:31.184 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1142,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635060224,"bytes_avail":192638468096,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":35303,"misplaced_total":72472,"misplaced_ratio":0.48712606247930235,"recovering_objects_per_sec":56,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":676,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:31.199 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:31.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:31.495 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:31.498 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:31.508 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":594,"stamp":"2024-09-18T14:25:30.211442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":35303,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620176,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123504,"statfs":{"total":193273528320,"available":192638468096,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1352,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":676,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003124"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":11437,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:26.335626+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:26.335626+0000","last_peered":"2024-09-18T14:25:26.335626+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:26.335626+0000","last_undegraded":"2024-09-18T14:25:26.335626+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4764,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2255,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2254},{"shards":"2,3","objects":2382}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":35303,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2256,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051592,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182409,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.434},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.46800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.67400000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.88800000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705898,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.59799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.628}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.39500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:34.510 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:34.842 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:34.845 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:34.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2024-09-18T14:25:34.212801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":34545,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624336,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188119344,"statfs":{"total":193273528320,"available":192634208256,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003459"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":12574,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:31.335002+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:31.335002+0000","last_peered":"2024-09-18T14:25:31.335002+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:31.335002+0000","last_undegraded":"2024-09-18T14:25:31.335002+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4006,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2634,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2633},{"shards":"2,3","objects":2003}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":34545,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051593,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182410,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.96499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.437},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59699999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314248,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057592,"statfs":{"total":96636764160,"available":96314974208,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.50900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.59199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:34.857 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:35.191 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:35.194 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:35.205 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2024-09-18T14:25:34.212801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":34545,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624336,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188119344,"statfs":{"total":193273528320,"available":192634208256,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003459"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":12574,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:31.335002+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:31.335002+0000","last_peered":"2024-09-18T14:25:31.335002+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:31.335002+0000","last_undegraded":"2024-09-18T14:25:31.335002+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4006,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2634,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2633},{"shards":"2,3","objects":2003}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":34545,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051593,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182410,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.96499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.437},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59699999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314248,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057592,"statfs":{"total":96636764160,"available":96314974208,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.50900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.59199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:35.206 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:35.206 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:35.206 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:35.206 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:35.206 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:35.206 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 12574, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:31.335002+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:31.335002+0000', 'last_peered': '2024-09-18T14:25:31.335002+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:31.335002+0000', 'last_undegraded': '2024-09-18T14:25:31.335002+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4006, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2634, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2633}, {'shards': '2,3', 'objects': 2003}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:35.207 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:35.208 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:35.584 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:35.584 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1146,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":639320064,"bytes_avail":192634208256,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":34545,"misplaced_total":72472,"misplaced_ratio":0.47666685064576664,"recovering_objects_per_sec":60,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:35.602 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:35.602 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:35.925 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:35.929 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:35.942 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":596,"stamp":"2024-09-18T14:25:34.212801+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":34545,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624336,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188119344,"statfs":{"total":193273528320,"available":192634208256,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003459"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":12574,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:31.335002+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:31.335002+0000","last_peered":"2024-09-18T14:25:31.335002+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:31.335002+0000","last_undegraded":"2024-09-18T14:25:31.335002+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4006,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2634,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2633},{"shards":"2,3","objects":2003}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":34545,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2635,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051593,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182410,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310088,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061752,"statfs":{"total":96636764160,"available":96319234048,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.96499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.437},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59699999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705899,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314248,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057592,"statfs":{"total":96636764160,"available":96314974208,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.50900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.64800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.59199999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:38.943 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:39.252 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:39.257 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:39.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2024-09-18T14:25:38.214068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":33695,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3060,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595216,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148464,"statfs":{"total":193273528320,"available":192664027136,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2306,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003554"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":13849,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:36.339082+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:36.339082+0000","last_peered":"2024-09-18T14:25:36.339082+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:36.339082+0000","last_undegraded":"2024-09-18T14:25:36.339082+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3156,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3058},{"shards":"2,3","objects":1578}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":33695,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3060,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051594,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.71499999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:39.271 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:39.593 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:39.597 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:39.609 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2024-09-18T14:25:38.214068+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":33695,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3060,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595216,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148464,"statfs":{"total":193273528320,"available":192664027136,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2306,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1153,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003554"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":13849,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:36.339082+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:36.339082+0000","last_peered":"2024-09-18T14:25:36.339082+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:36.339082+0000","last_undegraded":"2024-09-18T14:25:36.339082+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3156,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3058},{"shards":"2,3","objects":1578}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":33695,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3060,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051594,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.71499999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:39.610 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:39.610 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:39.610 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:39.611 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 13849, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:36.339082+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:36.339082+0000', 'last_peered': '2024-09-18T14:25:36.339082+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:36.339082+0000', 'last_undegraded': '2024-09-18T14:25:36.339082+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3156, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3059, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3058}, {'shards': '2,3', 'objects': 1578}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:39.612 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:39.612 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:39.612 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:39.612 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:39.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:40.012 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:40.012 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1150,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609501184,"bytes_avail":192664027136,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":33695,"misplaced_total":72472,"misplaced_ratio":0.46493818302240864,"recovering_objects_per_sec":96,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1153,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:40.028 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:40.029 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:40.345 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:40.348 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:40.358 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":599,"stamp":"2024-09-18T14:25:40.214502+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":33695,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3060,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595216,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148464,"statfs":{"total":193273528320,"available":192664027136,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1608,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":804,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003488"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":13849,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:36.339082+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:36.339082+0000","last_peered":"2024-09-18T14:25:36.339082+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:36.339082+0000","last_undegraded":"2024-09-18T14:25:36.339082+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3156,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3059,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3058},{"shards":"2,3","objects":1578}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":33695,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3060,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051594,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182411,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.68400000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.71499999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:43.361 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:43.673 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:43.677 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:43.690 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2024-09-18T14:25:42.215196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":32707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595216,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148464,"statfs":{"total":193273528320,"available":192664027136,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2596,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003754"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":15331,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:41.342510+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:41.342510+0000","last_peered":"2024-09-18T14:25:41.342510+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:41.342510+0000","last_undegraded":"2024-09-18T14:25:41.342510+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2168,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3552},{"shards":"2,3","objects":1084}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":32707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051594,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182412,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.69299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:43.691 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:44.001 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:44.015 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":600,"stamp":"2024-09-18T14:25:42.215196+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":32707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595216,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148464,"statfs":{"total":193273528320,"available":192664027136,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2596,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1298,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003754"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":15331,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:41.342510+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:41.342510+0000","last_peered":"2024-09-18T14:25:41.342510+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:41.342510+0000","last_undegraded":"2024-09-18T14:25:41.342510+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2168,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3552},{"shards":"2,3","objects":1084}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":32707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051594,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182412,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.69299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705900,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.51200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.53900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:44.016 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:44.016 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:44.016 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:44.016 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:44.016 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:44.016 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:44.016 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:44.016 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:44.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:44.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:44.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:44.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 15331, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:41.342510+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:41.342510+0000', 'last_peered': '2024-09-18T14:25:41.342510+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:41.342510+0000', 'last_undegraded': '2024-09-18T14:25:41.342510+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2168, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3553, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3552}, {'shards': '2,3', 'objects': 1084}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:44.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:44.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:44.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:44.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:44.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:44.414 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:44.414 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1155,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609501184,"bytes_avail":192664027136,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":32707,"misplaced_total":72472,"misplaced_ratio":0.45130533171431719,"recovering_objects_per_sec":108,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1298,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":10,"modified":"2024-09-18T14:22:38.167822+0000","services":{}},"progress_events":{}} 2024-09-18T14:25:44.430 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:44.430 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:44.748 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:44.751 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:44.767 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":601,"stamp":"2024-09-18T14:25:44.215702+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":32707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595216,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148464,"statfs":{"total":193273528320,"available":192664027136,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1838,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":919,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003529"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":15331,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:41.342510+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:41.342510+0000","last_peered":"2024-09-18T14:25:41.342510+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:41.342510+0000","last_undegraded":"2024-09-18T14:25:41.342510+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2168,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3552},{"shards":"2,3","objects":1084}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":32707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051595,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182412,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.69299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.499},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.70799999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.621},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.82599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:47.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:48.094 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:48.097 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:48.111 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":602,"stamp":"2024-09-18T14:25:46.216248+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":32707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595216,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148464,"statfs":{"total":193273528320,"available":192664027136,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1838,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":919,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003447"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":15331,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:41.342510+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:41.342510+0000","last_peered":"2024-09-18T14:25:41.342510+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:41.342510+0000","last_undegraded":"2024-09-18T14:25:41.342510+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2168,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3553,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3552},{"shards":"2,3","objects":1084}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":32707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3554,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051595,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182412,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.39600000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.67000000000000004,"15min":0.65900000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":6.8170000000000002,"5min":6.8170000000000002,"15min":6.8170000000000002},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.71699999999999997,"15min":0.71299999999999997},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":4.6189999999999998,"5min":4.6189999999999998,"15min":4.6189999999999998},"last":0.69299999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.59099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705901,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297608,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074232,"statfs":{"total":96636764160,"available":96332013568,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.499},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.70799999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68899999999999995,"5min":0.84199999999999997,"15min":1.0029999999999999},"last":0.621},{"interface":"front","average":{"1min":0.67900000000000005,"5min":0.77000000000000002,"15min":0.77200000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"last":0.42299999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.82599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:48.112 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:48.428 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:48.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:48.444 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":603,"stamp":"2024-09-18T14:25:48.216720+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603536,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140144,"statfs":{"total":193273528320,"available":192655507456,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003278"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":17236,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:46.346467+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:46.346467+0000","last_peered":"2024-09-18T14:25:46.346467+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:46.346467+0000","last_undegraded":"2024-09-18T14:25:46.346467+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":898,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4188,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4187},{"shards":"2,3","objects":449}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051596,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705903,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301768,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070072,"statfs":{"total":96636764160,"available":96327753728,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.373},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.45400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.83599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301768,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070072,"statfs":{"total":96636764160,"available":96327753728,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:48.445 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:48.445 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4630, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416034+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:24:58.416034+0000', 'last_peered': '2024-09-18T14:24:58.416034+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:24:58.416034+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:48.445 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:48.445 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:48.445 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:48.445 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4496, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415815+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:24:58.415815+0000', 'last_peered': '2024-09-18T14:24:58.415815+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:24:58.415815+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:48.445 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:48.445 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:48.446 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:48.446 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4528, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.415621+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:24:58.415621+0000', 'last_peered': '2024-09-18T14:24:58.415621+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:24:58.415621+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:48.446 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:48.446 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 17236, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:46.346467+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:46.346467+0000', 'last_peered': '2024-09-18T14:25:46.346467+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:46.346467+0000', 'last_undegraded': '2024-09-18T14:25:46.346467+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 898, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4188, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 4187}, {'shards': '2,3', 'objects': 449}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:48.446 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:48.446 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:48.446 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:48.446 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4565, 'reported_epoch': 30, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:58.416496+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:24:58.416496+0000', 'last_peered': '2024-09-18T14:24:58.416496+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:24:58.416496+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:48.447 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:48.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:48.826 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1159,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":31,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":8},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618020864,"bytes_avail":192655507456,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":31437,"misplaced_total":72472,"misplaced_ratio":0.43378132244177059,"recovering_objects_per_sec":129,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1554,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-18T14:25:48.216827+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T14:25:48.842 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:48.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:49.161 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:49.164 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:49.174 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":604,"stamp":"2024-09-18T14:25:49.043516+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603536,"kb_used_data":512,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188140144,"statfs":{"total":193273528320,"available":192655507456,"internally_reserved":0,"allocated":524288,"data_stored":185851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2258,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1129,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.829447"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4630,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416034+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:24:58.416034+0000","last_peered":"2024-09-18T14:24:58.416034+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:24:58.416034+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4496,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415815+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:24:58.415815+0000","last_peered":"2024-09-18T14:24:58.415815+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:24:58.415815+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.415621+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:24:58.415621+0000","last_peered":"2024-09-18T14:24:58.415621+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:24:58.415621+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":17236,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:46.346467+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:46.346467+0000","last_peered":"2024-09-18T14:25:46.346467+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:46.346467+0000","last_undegraded":"2024-09-18T14:25:46.346467+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":898,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4188,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4187},{"shards":"2,3","objects":449}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":30,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:58.416496+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:24:58.416496+0000","last_peered":"2024-09-18T14:24:58.416496+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:24:58.416496+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":498532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051596,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182413,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705903,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301768,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070072,"statfs":{"total":96636764160,"available":96327753728,"internally_reserved":0,"allocated":262144,"data_stored":93055,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.373},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.45400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.83599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301768,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070072,"statfs":{"total":96636764160,"available":96327753728,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:52.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:52.487 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:52.501 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2024-09-18T14:25:52.217886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":607744,"kb_used_data":560,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135936,"statfs":{"total":193273528320,"available":192651198464,"internally_reserved":0,"allocated":573440,"data_stored":218818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1270,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002184"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":17236,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:46.346467+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:46.346467+0000","last_peered":"2024-09-18T14:25:46.346467+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:46.346467+0000","last_undegraded":"2024-09-18T14:25:46.346467+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":898,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4188,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4187},{"shards":"2,3","objects":449}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":524764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051596,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.55500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301768,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070072,"statfs":{"total":96636764160,"available":96327753728,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:52.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:52.785 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:52.788 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:52.798 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2024-09-18T14:25:52.217886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":607744,"kb_used_data":560,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135936,"statfs":{"total":193273528320,"available":192651198464,"internally_reserved":0,"allocated":573440,"data_stored":218818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1270,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002184"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":17236,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:46.346467+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:46.346467+0000","last_peered":"2024-09-18T14:25:46.346467+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:46.346467+0000","last_undegraded":"2024-09-18T14:25:46.346467+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":898,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4188,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4187},{"shards":"2,3","objects":449}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":524764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051596,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.55500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301768,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070072,"statfs":{"total":96636764160,"available":96327753728,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:52.799 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:52.799 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:52.799 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:52.799 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4599, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.548265+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:24:59.548265+0000', 'last_peered': '2024-09-18T14:24:59.548265+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:24:59.548265+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:52.799 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4554, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.505408+0000', 'last_change': '2024-09-18T14:24:59.528416+0000', 'last_active': '2024-09-18T14:24:59.505408+0000', 'last_peered': '2024-09-18T14:24:59.505408+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:24:59.505408+0000', 'last_undegraded': '2024-09-18T14:24:59.491215+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 3442, 'num_objects_misplaced': 3442, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3442}, {'shards': '3', 'objects': 1061}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:PG 1.2 is not active+clean 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.2', 'version': "17'4637", 'reported_seq': 17236, 'reported_epoch': 31, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:46.346467+0000', 'last_change': '2024-09-18T14:24:57.527057+0000', 'last_active': '2024-09-18T14:25:46.346467+0000', 'last_peered': '2024-09-18T14:25:46.346467+0000', 'last_clean': '2024-09-18T14:23:59.155252+0000', 'last_became_active': '2024-09-18T14:24:48.411418+0000', 'last_became_peered': '2024-09-18T14:24:48.411418+0000', 'last_unstale': '2024-09-18T14:25:46.346467+0000', 'last_undegraded': '2024-09-18T14:25:46.346467+0000', 'last_fullsized': '2024-09-18T14:24:48.309051+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4637, 'log_dups_size': 0, 'ondisk_log_size': 4637, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000794326, 'stat_sum': {'num_bytes': 30884, 'num_objects': 4636, 'num_object_clones': 0, 'num_object_copies': 9272, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 898, 'num_objects_unfound': 0, 'num_objects_dirty': 4636, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4637, 'num_write_kb': 4661, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4188, 'num_bytes_recovered': 38, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 4187}, {'shards': '2,3', 'objects': 449}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4541, 'reported_epoch': 31, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:24:59.565667+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:24:59.565667+0000', 'last_peered': '2024-09-18T14:24:59.565667+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:24:59.565667+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:52.800 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:52.801 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:52.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:53.151 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:53.151 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 30539/72472 objects degraded (42.139%), 7 pgs degraded, 5 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1164,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":622329856,"bytes_avail":192651198464,"bytes_total":193273528320,"degraded_objects":30539,"degraded_total":72472,"degraded_ratio":0.42139033005850535,"misplaced_objects":31437,"misplaced_total":72472,"misplaced_ratio":0.43378132244177059,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":635,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-18T14:25:48.216827+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T14:25:53.169 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:53.169 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:53.464 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:53.467 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:53.477 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":607,"stamp":"2024-09-18T14:25:52.217886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":607744,"kb_used_data":560,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135936,"statfs":{"total":193273528320,"available":192651198464,"internally_reserved":0,"allocated":573440,"data_stored":218818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1270,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":635,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002184"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4599,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.548265+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:24:59.548265+0000","last_peered":"2024-09-18T14:24:59.548265+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:24:59.548265+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4554,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.505408+0000","last_change":"2024-09-18T14:24:59.528416+0000","last_active":"2024-09-18T14:24:59.505408+0000","last_peered":"2024-09-18T14:24:59.505408+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:24:59.505408+0000","last_undegraded":"2024-09-18T14:24:59.491215+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3442,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3442},{"shards":"3","objects":1061}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":17236,"reported_epoch":31,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:46.346467+0000","last_change":"2024-09-18T14:24:57.527057+0000","last_active":"2024-09-18T14:25:46.346467+0000","last_peered":"2024-09-18T14:25:46.346467+0000","last_clean":"2024-09-18T14:23:59.155252+0000","last_became_active":"2024-09-18T14:24:48.411418+0000","last_became_peered":"2024-09-18T14:24:48.411418+0000","last_unstale":"2024-09-18T14:25:46.346467+0000","last_undegraded":"2024-09-18T14:25:46.346467+0000","last_fullsized":"2024-09-18T14:24:48.309051+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:11:22.469120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":898,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4188,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4187},{"shards":"2,3","objects":449}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4541,"reported_epoch":31,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:24:59.565667+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:24:59.565667+0000","last_peered":"2024-09-18T14:24:59.565667+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:24:59.565667+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":30539,"num_objects_misplaced":31437,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4189,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":679936,"data_stored":524764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":8,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051596,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.55500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705902,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301768,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070072,"statfs":{"total":96636764160,"available":96327753728,"internally_reserved":0,"allocated":262144,"data_stored":92796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:56.479 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:56.787 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:56.790 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:56.801 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-09-18T14:25:56.219205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":33529,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4864,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3442,"num_objects_misplaced":2092,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002485"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":5235,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:52.434655+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:25:52.434655+0000","last_peered":"2024-09-18T14:25:52.434655+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:25:52.434655+0000","last_undegraded":"2024-09-18T14:25:52.434655+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6432,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3216},{"shards":"3","objects":1287}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":33529,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4864,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051597,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.55500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:56.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:57.119 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:57.121 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:57.131 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-09-18T14:25:56.219205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":33529,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4864,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3442,"num_objects_misplaced":2092,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002485"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":5235,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:52.434655+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:25:52.434655+0000","last_peered":"2024-09-18T14:25:52.434655+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:25:52.434655+0000","last_undegraded":"2024-09-18T14:25:52.434655+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6432,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3216},{"shards":"3","objects":1287}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":33529,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4864,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051597,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.55500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:25:57.132 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:25:57.132 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:57.132 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:25:57.132 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:57.132 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:25:57.132 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:57.133 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:25:57.133 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 5235, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:52.434655+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:25:52.434655+0000', 'last_peered': '2024-09-18T14:25:52.434655+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:25:52.434655+0000', 'last_undegraded': '2024-09-18T14:25:52.434655+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6432, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 226, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3216}, {'shards': '3', 'objects': 1287}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:57.133 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:25:57.133 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:57.133 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:25:57.133 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:25:57.133 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:25:57.133 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:25:57.134 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:25:57.513 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:57.514 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 4 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1168,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626638848,"bytes_avail":192646889472,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":33529,"misplaced_total":72472,"misplaced_ratio":0.46264764322772933,"recovering_objects_per_sec":84,"recovering_bytes_per_sec":3277,"recovering_keys_per_sec":0,"num_objects_recovered":675,"num_bytes_recovered":26232,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":11,"modified":"2024-09-18T14:25:48.216827+0000","services":{"osd":{"daemons":{"summary":"","3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T14:25:57.531 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:25:57.531 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:25:57.821 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:25:57.823 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:25:57.837 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":609,"stamp":"2024-09-18T14:25:56.219205+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":33529,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4864,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3442,"num_objects_misplaced":2092,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":675,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002485"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":5235,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:52.434655+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:25:52.434655+0000","last_peered":"2024-09-18T14:25:52.434655+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:25:52.434655+0000","last_undegraded":"2024-09-18T14:25:52.434655+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6432,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":226,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3216},{"shards":"3","objects":1287}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":33529,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4864,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051597,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182414,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.51800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.55500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705903,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.34999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:24:16 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61099999999999999,"15min":0.624},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":1.0609999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"last":0.44800000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:00.839 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:01.157 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:01.162 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:01.174 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2024-09-18T14:26:00.220552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32893,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5182,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3442,"num_objects_misplaced":1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.174882"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":6189,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:57.433973+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:25:57.433973+0000","last_peered":"2024-09-18T14:25:57.433973+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:25:57.433973+0000","last_undegraded":"2024-09-18T14:25:57.433973+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5796,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2898},{"shards":"3","objects":1605}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32893,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5182,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051598,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182415,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.42399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.68600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:01.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:01.502 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:01.505 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:01.521 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":611,"stamp":"2024-09-18T14:26:00.220552+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32893,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5182,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3442,"num_objects_misplaced":1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.174882"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":6189,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:57.433973+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:25:57.433973+0000","last_peered":"2024-09-18T14:25:57.433973+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:25:57.433973+0000","last_undegraded":"2024-09-18T14:25:57.433973+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5796,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2898},{"shards":"3","objects":1605}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32893,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5182,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051598,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182415,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.57899999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.48399999999999999},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.42399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:24:02 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.67200000000000004,"15min":0.69399999999999995},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.80500000000000005,"5min":0.89500000000000002,"15min":1.0429999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.68300000000000005,"15min":0.72499999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.29799999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":1.323},"last":0.68600000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 6189, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:25:57.433973+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:25:57.433973+0000', 'last_peered': '2024-09-18T14:25:57.433973+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:25:57.433973+0000', 'last_undegraded': '2024-09-18T14:25:57.433973+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5796, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 544, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2898}, {'shards': '3', 'objects': 1605}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:01.522 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:01.523 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:01.523 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:01.523 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:01.523 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:01.523 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:01.908 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:01.909 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1172,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626638848,"bytes_avail":192646889472,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":32893,"misplaced_total":72472,"misplaced_ratio":0.45387184015895793,"recovering_objects_per_sec":97,"recovering_bytes_per_sec":2578,"recovering_keys_per_sec":0,"num_objects_recovered":993,"num_bytes_recovered":26232,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:01.924 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:01.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:02.230 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:02.233 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:02.244 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":612,"stamp":"2024-09-18T14:26:02.220976+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32893,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5182,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3442,"num_objects_misplaced":1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":993,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003723"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":6189,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:25:57.433973+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:25:57.433973+0000","last_peered":"2024-09-18T14:25:57.433973+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:25:57.433973+0000","last_undegraded":"2024-09-18T14:25:57.433973+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5796,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":544,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2898},{"shards":"3","objects":1605}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32893,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5182,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051598,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.627},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.499}]}]},{"osd":0,"up_from":9,"seq":38654705904,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.64900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.51700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:05.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:05.561 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:05.565 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:05.577 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2024-09-18T14:26:04.221903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32209,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5524,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3442,"num_objects_misplaced":772,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1335,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004016"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":7215,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:02.446855+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:02.446855+0000","last_peered":"2024-09-18T14:26:02.446855+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:02.446855+0000","last_undegraded":"2024-09-18T14:26:02.446855+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5112,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":886,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2556},{"shards":"3","objects":1947}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32209,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5524,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051599,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.627},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.499}]}]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.45100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.77900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:05.578 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:05.876 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:05.880 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:05.893 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":613,"stamp":"2024-09-18T14:26:04.221903+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32209,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5524,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3442,"num_objects_misplaced":772,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1335,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004016"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":7215,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:02.446855+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:02.446855+0000","last_peered":"2024-09-18T14:26:02.446855+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:02.446855+0000","last_undegraded":"2024-09-18T14:26:02.446855+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5112,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":886,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2556},{"shards":"3","objects":1947}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32209,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5524,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051599,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.627},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.499}]}]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.45100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.77900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:05.894 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:05.894 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:05.894 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:05.894 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:05.894 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:05.894 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:05.895 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:05.895 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 7215, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:02.446855+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:02.446855+0000', 'last_peered': '2024-09-18T14:26:02.446855+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:02.446855+0000', 'last_undegraded': '2024-09-18T14:26:02.446855+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5112, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 886, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2556}, {'shards': '3', 'objects': 1947}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:05.895 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:05.895 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:05.895 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:05.895 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:05.895 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:05.895 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:05.896 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:06.271 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:06.271 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1177,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626638848,"bytes_avail":192646889472,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":32209,"misplaced_total":72472,"misplaced_ratio":0.44443371233027934,"recovering_objects_per_sec":111,"recovering_bytes_per_sec":2185,"recovering_keys_per_sec":0,"num_objects_recovered":1335,"num_bytes_recovered":26232,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:06.288 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:06.288 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:06.588 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:06.603 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":614,"stamp":"2024-09-18T14:26:06.222758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32209,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5524,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":611952,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131728,"statfs":{"total":193273528320,"available":192646889472,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004208"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":7215,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:02.446855+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:02.446855+0000","last_peered":"2024-09-18T14:26:02.446855+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:02.446855+0000","last_undegraded":"2024-09-18T14:26:02.446855+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5112,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":886,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2556},{"shards":"3","objects":1947}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":32209,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5524,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051599,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182416,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.40500000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.627},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.48699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.499}]}]},{"osd":0,"up_from":9,"seq":38654705905,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305976,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065864,"statfs":{"total":96636764160,"available":96323444736,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.45100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.77900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.60199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:09.605 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:09.923 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:09.936 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":615,"stamp":"2024-09-18T14:26:08.223335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":31479,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5889,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2050,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1025,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004130"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":8310,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:07.438147+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:07.438147+0000","last_peered":"2024-09-18T14:26:07.438147+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:07.438147+0000","last_undegraded":"2024-09-18T14:26:07.438147+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4382,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1251,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2191},{"shards":"3","objects":2312}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":31479,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5889,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051600,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.39300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.317},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.435},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:09.937 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:10.249 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:10.267 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2024-09-18T14:26:10.223987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":31479,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5889,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1414,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004173"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":8310,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:07.438147+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:07.438147+0000","last_peered":"2024-09-18T14:26:07.438147+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:07.438147+0000","last_undegraded":"2024-09-18T14:26:07.438147+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4382,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1251,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2191},{"shards":"3","objects":2312}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":31479,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5889,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051600,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.39300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.317},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.435},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:10.268 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:10.268 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:10.268 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:10.268 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:10.268 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:10.268 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:10.268 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:10.268 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 8310, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:07.438147+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:07.438147+0000', 'last_peered': '2024-09-18T14:26:07.438147+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:07.438147+0000', 'last_undegraded': '2024-09-18T14:26:07.438147+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4382, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1251, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2191}, {'shards': '3', 'objects': 2312}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:10.269 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:10.269 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:10.269 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:10.269 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:10.269 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:10.270 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:10.270 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:10.649 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:10.649 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1181,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635158528,"bytes_avail":192638369792,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":31479,"misplaced_total":72472,"misplaced_ratio":0.43436085660668949,"recovering_objects_per_sec":58,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":707,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:10.665 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:10.666 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:10.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:10.983 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:10.995 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":616,"stamp":"2024-09-18T14:26:10.223987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":31479,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5889,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1414,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":707,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004173"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":8310,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:07.438147+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:07.438147+0000","last_peered":"2024-09-18T14:26:07.438147+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:07.438147+0000","last_undegraded":"2024-09-18T14:26:07.438147+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4382,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1251,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2191},{"shards":"3","objects":2312}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":31479,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5889,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051600,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182417,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:25:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.57299999999999995,"15min":0.56299999999999994},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.61299999999999999,"5min":6.758,"15min":6.758},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.64600000000000002,"15min":0.62},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":1.6399999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.39300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705906,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.317},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.751}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.435},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:13.997 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:14.311 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:14.314 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:14.325 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-09-18T14:26:14.225131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30671,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2222,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004155"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":9522,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:12.442544+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:12.442544+0000","last_peered":"2024-09-18T14:26:12.442544+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:12.442544+0000","last_undegraded":"2024-09-18T14:26:12.442544+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3574,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1787},{"shards":"3","objects":2716}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30671,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051601,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182418,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.442},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.56000000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.746}]}]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.61099999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.92100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:14.326 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:14.635 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:14.655 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-09-18T14:26:14.225131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30671,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2222,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004155"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":9522,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:12.442544+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:12.442544+0000","last_peered":"2024-09-18T14:26:12.442544+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:12.442544+0000","last_undegraded":"2024-09-18T14:26:12.442544+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3574,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1787},{"shards":"3","objects":2716}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30671,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051601,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182418,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.442},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.56000000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.746}]}]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.61099999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.92100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:14.656 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:14.656 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:14.656 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:14.656 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:14.656 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 9522, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:12.442544+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:12.442544+0000', 'last_peered': '2024-09-18T14:26:12.442544+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:12.442544+0000', 'last_undegraded': '2024-09-18T14:26:12.442544+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3574, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1655, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1787}, {'shards': '3', 'objects': 2716}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:14.657 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:14.658 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:15.023 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:15.024 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1185,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635158528,"bytes_avail":192638369792,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":30671,"misplaced_total":72472,"misplaced_ratio":0.42321172314825034,"recovering_objects_per_sec":92,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:15.041 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:15.041 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:15.345 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:15.348 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:15.364 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":618,"stamp":"2024-09-18T14:26:14.225131+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30671,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2222,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1111,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004155"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":9522,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:12.442544+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:12.442544+0000","last_peered":"2024-09-18T14:26:12.442544+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:12.442544+0000","last_undegraded":"2024-09-18T14:26:12.442544+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3574,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1655,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1787},{"shards":"3","objects":2716}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30671,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6293,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051601,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182418,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.442},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.35199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.56000000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.746}]}]},{"osd":0,"up_from":9,"seq":38654705907,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.88600000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.61099999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.92100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":1.0189999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.77000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:18.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:18.667 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:18.670 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:18.681 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2024-09-18T14:26:18.226327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2088,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003569"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182419,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.499},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.93100000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.309},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:18.683 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:18.966 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:18.969 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:18.979 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2024-09-18T14:26:18.226327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2088,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003569"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182419,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.499},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.93100000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.309},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:18.980 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:18.980 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:18.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:18.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:18.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:18.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:18.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:18.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 10348, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:15.479707+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:15.479707+0000', 'last_peered': '2024-09-18T14:26:15.479707+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:15.479707+0000', 'last_undegraded': '2024-09-18T14:26:15.479707+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3024, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1930, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1512}, {'shards': '3', 'objects': 2991}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:18.982 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:18.982 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:18.982 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:18.982 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:18.982 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:18.982 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:18.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:19.329 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:19.329 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1190,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635158528,"bytes_avail":192638369792,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":30121,"misplaced_total":72472,"misplaced_ratio":0.41562258527431284,"recovering_objects_per_sec":86,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1044,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:19.342 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:19.343 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:19.627 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:19.630 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:19.641 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":620,"stamp":"2024-09-18T14:26:18.226327+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2088,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1044,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003569"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182419,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.499},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.93100000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.309},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:22.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:22.938 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:22.942 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:22.955 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":622,"stamp":"2024-09-18T14:26:22.227681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1358,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.94099999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.309},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:22.956 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:23.243 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:23.246 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:23.260 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":622,"stamp":"2024-09-18T14:26:22.227681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1358,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.94099999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.309},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:23.261 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:23.261 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:23.261 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:23.261 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 10348, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:15.479707+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:15.479707+0000', 'last_peered': '2024-09-18T14:26:15.479707+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:15.479707+0000', 'last_undegraded': '2024-09-18T14:26:15.479707+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3024, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1930, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1512}, {'shards': '3', 'objects': 2991}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:23.262 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:23.263 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:23.605 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:23.606 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1194,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635158528,"bytes_avail":192638369792,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":30121,"misplaced_total":72472,"misplaced_ratio":0.41562258527431284,"recovering_objects_per_sec":56,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":679,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:23.624 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:23.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:23.906 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:23.909 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:23.919 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":622,"stamp":"2024-09-18T14:26:22.227681+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1358,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":679,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003694"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051602,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.94099999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705908,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:25:17 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.54500000000000004,"15min":0.55200000000000005},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.73499999999999999,"5min":0.84999999999999998,"15min":0.95299999999999996},"last":0.309},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.63400000000000001,"15min":0.67500000000000004},"min":{"1min":0.35099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78800000000000003,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.50800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.61099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:26.920 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:27.216 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:27.220 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:27.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2024-09-18T14:26:26.229217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-550,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004086"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051603,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.94099999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.94799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.65700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:27.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:27.522 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:27.525 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:27.539 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2024-09-18T14:26:26.229217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-550,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004086"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051603,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.94099999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.94799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.65700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:27.540 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 10348, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:15.479707+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:15.479707+0000', 'last_peered': '2024-09-18T14:26:15.479707+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:15.479707+0000', 'last_undegraded': '2024-09-18T14:26:15.479707+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3024, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1930, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1512}, {'shards': '3', 'objects': 2991}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:27.541 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:27.542 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:27.542 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:27.542 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:27.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:27.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:27.889 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1198,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635158528,"bytes_avail":192638369792,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":30121,"misplaced_total":72472,"misplaced_ratio":0.41562258527431284,"recovering_objects_per_sec":22,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":275,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:27.903 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:27.903 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:28.188 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:28.202 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":624,"stamp":"2024-09-18T14:26:26.229217+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-550,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004086"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051603,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182420,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.61199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.94099999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705909,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.94799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.65700000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.74399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:31.204 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:31.531 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:31.534 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:31.549 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2024-09-18T14:26:30.230715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004387"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051604,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182421,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705911,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.71099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.626},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.74099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:31.550 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:31.885 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:31.890 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:31.903 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":626,"stamp":"2024-09-18T14:26:30.230715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004387"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051604,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182421,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705911,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.71099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.626},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.74099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 10348, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:15.479707+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:15.479707+0000', 'last_peered': '2024-09-18T14:26:15.479707+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:15.479707+0000', 'last_undegraded': '2024-09-18T14:26:15.479707+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3024, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1930, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1512}, {'shards': '3', 'objects': 2991}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:31.904 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:31.905 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:31.905 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:31.905 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:31.905 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:31.905 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:31.905 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:32.276 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:32.277 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1203,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635158528,"bytes_avail":192638369792,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":30121,"misplaced_total":72472,"misplaced_ratio":0.41562258527431284},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:32.290 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:26:32.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:32.612 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:32.615 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:32.626 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":627,"stamp":"2024-09-18T14:26:32.231311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620272,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123408,"statfs":{"total":193273528320,"available":192638369792,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004276"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10348,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:15.479707+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:15.479707+0000","last_peered":"2024-09-18T14:26:15.479707+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:15.479707+0000","last_undegraded":"2024-09-18T14:26:15.479707+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3024,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1512},{"shards":"3","objects":2991}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":30121,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6568,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051604,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182422,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.89600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.84099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705910,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.626},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.69999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.47099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.74099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:35.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:35.956 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:35.969 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":628,"stamp":"2024-09-18T14:26:34.231758+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":29739,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6759,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624432,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188119248,"statfs":{"total":193273528320,"available":192634109952,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-382,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004076"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10920,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:32.445254+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:32.445254+0000","last_peered":"2024-09-18T14:26:32.445254+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:32.445254+0000","last_undegraded":"2024-09-18T14:26:32.445254+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2642,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1321},{"shards":"3","objects":3182}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":29739,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6759,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051605,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182422,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.89600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.84099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314296,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057544,"statfs":{"total":96636764160,"available":96314925056,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.79200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.72599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:35.971 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:36.286 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:36.294 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:36.307 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":629,"stamp":"2024-09-18T14:26:36.232509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":29739,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6759,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624432,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188119248,"statfs":{"total":193273528320,"available":192634109952,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-382,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004111"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10920,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:32.445254+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:32.445254+0000","last_peered":"2024-09-18T14:26:32.445254+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:32.445254+0000","last_undegraded":"2024-09-18T14:26:32.445254+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2642,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1321},{"shards":"3","objects":3182}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":29739,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6759,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051605,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182422,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.89600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.84099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314296,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057544,"statfs":{"total":96636764160,"available":96314925056,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.79200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.72599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:36.308 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:36.308 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:36.308 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:36.308 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:36.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:36.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:36.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:36.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 10920, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:32.445254+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:32.445254+0000', 'last_peered': '2024-09-18T14:26:32.445254+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:32.445254+0000', 'last_undegraded': '2024-09-18T14:26:32.445254+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2642, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2121, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1321}, {'shards': '3', 'objects': 3182}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:36.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:36.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:36.310 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:36.310 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:36.310 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:36.310 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:36.311 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:36.691 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:36.692 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1207,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":639418368,"bytes_avail":192634109952,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":29739,"misplaced_total":72472,"misplaced_ratio":0.41035158406005079,"recovering_objects_per_sec":15,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:36.705 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:36.705 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:37.009 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:37.013 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:37.026 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":629,"stamp":"2024-09-18T14:26:36.232509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":29739,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6759,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624432,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188119248,"statfs":{"total":193273528320,"available":192634109952,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-382,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":191,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004111"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":10920,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:32.445254+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:32.445254+0000","last_peered":"2024-09-18T14:26:32.445254+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:32.445254+0000","last_undegraded":"2024-09-18T14:26:32.445254+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2642,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1321},{"shards":"3","objects":3182}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":29739,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6759,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051605,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182422,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310136,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061704,"statfs":{"total":96636764160,"available":96319184896,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.78400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.89600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.84099999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705911,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314296,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057544,"statfs":{"total":96636764160,"available":96314925056,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.79200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.72599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:40.028 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:40.358 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:40.361 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:40.372 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2024-09-18T14:26:40.233806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":28607,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7325,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595312,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148368,"statfs":{"total":193273528320,"available":192663928832,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1514,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003799"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":12618,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:37.448495+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:37.448495+0000","last_peered":"2024-09-18T14:26:37.448495+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:37.448495+0000","last_undegraded":"2024-09-18T14:26:37.448495+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1510,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2687,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":755},{"shards":"3","objects":3748}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":28607,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7325,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051606,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705913,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.439}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:40.373 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:40.689 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:40.707 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2024-09-18T14:26:40.233806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":28607,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7325,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595312,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148368,"statfs":{"total":193273528320,"available":192663928832,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1514,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003799"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":12618,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:37.448495+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:37.448495+0000","last_peered":"2024-09-18T14:26:37.448495+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:37.448495+0000","last_undegraded":"2024-09-18T14:26:37.448495+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1510,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2687,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":755},{"shards":"3","objects":3748}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":28607,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7325,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051606,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705913,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.439}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:40.708 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4601, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050213+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:49.050213+0000', 'last_peered': '2024-09-18T14:25:49.050213+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:49.050213+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 12618, 'reported_epoch': 33, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:37.448495+0000', 'last_change': '2024-09-18T14:25:48.716952+0000', 'last_active': '2024-09-18T14:26:37.448495+0000', 'last_peered': '2024-09-18T14:26:37.448495+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:24:59.500465+0000', 'last_became_peered': '2024-09-18T14:24:59.500465+0000', 'last_unstale': '2024-09-18T14:26:37.448495+0000', 'last_undegraded': '2024-09-18T14:26:37.448495+0000', 'last_fullsized': '2024-09-18T14:24:59.417326+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1510, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2687, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 755}, {'shards': '3', 'objects': 3748}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:40.709 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:40.710 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4543, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.050392+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:49.050392+0000', 'last_peered': '2024-09-18T14:25:49.050392+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:49.050392+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:40.710 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:40.710 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:40.710 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:41.098 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:41.099 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 7 pgs undersized","count":13},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1211,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":33,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":7},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609599488,"bytes_avail":192663928832,"bytes_total":193273528320,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":28607,"misplaced_total":72472,"misplaced_ratio":0.39473175847223757,"recovering_objects_per_sec":63,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":757,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:41.112 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:41.113 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:41.436 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:41.439 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:41.448 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":631,"stamp":"2024-09-18T14:26:40.233806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":28607,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7325,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595312,"kb_used_data":608,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148368,"statfs":{"total":193273528320,"available":192663928832,"internally_reserved":0,"allocated":622592,"data_stored":251785,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1514,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":757,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003799"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4601,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050213+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:49.050213+0000","last_peered":"2024-09-18T14:25:49.050213+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:49.050213+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":12618,"reported_epoch":33,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:37.448495+0000","last_change":"2024-09-18T14:25:48.716952+0000","last_active":"2024-09-18T14:26:37.448495+0000","last_peered":"2024-09-18T14:26:37.448495+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:24:59.500465+0000","last_became_peered":"2024-09-18T14:24:59.500465+0000","last_unstale":"2024-09-18T14:26:37.448495+0000","last_undegraded":"2024-09-18T14:26:37.448495+0000","last_fullsized":"2024-09-18T14:24:59.417326+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:21:25.925399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1510,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2687,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":755},{"shards":"3","objects":3748}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4672,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065230+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065230+0000","last_peered":"2024-09-18T14:25:50.065230+0000","last_clean":"2024-09-18T14:25:50.065230+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065230+0000","last_undegraded":"2024-09-18T14:25:50.065230+0000","last_fullsized":"2024-09-18T14:25:50.065230+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4543,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.050392+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:49.050392+0000","last_peered":"2024-09-18T14:25:49.050392+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:49.050392+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":28607,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7325,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":712704,"data_stored":550996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":9,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051606,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182423,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705913,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.439}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.64200000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705912,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297656,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074184,"statfs":{"total":96636764160,"available":96331964416,"internally_reserved":0,"allocated":311296,"data_stored":125763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.45700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.60499999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":26251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:44.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:44.767 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:44.769 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:44.780 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":635,"stamp":"2024-09-18T14:26:44.235126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":27097,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603708,"kb_used_data":680,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139972,"statfs":{"total":193273528320,"available":192655331328,"internally_reserved":0,"allocated":696320,"data_stored":307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2642,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002616"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:50.062116+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:50.062116+0000","last_peered":"2024-09-18T14:25:50.062116+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:50.062116+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4553,"reported_epoch":34,"state":"peering","last_fresh":"2024-09-18T14:26:42.314650+0000","last_change":"2024-09-18T14:26:42.314650+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:26:42.314650+0000","last_undegraded":"2024-09-18T14:26:42.314650+0000","last_fullsized":"2024-09-18T14:26:42.314650+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:39.400748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4673,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065831+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065831+0000","last_peered":"2024-09-18T14:25:50.065831+0000","last_clean":"2024-09-18T14:25:50.065831+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065831+0000","last_undegraded":"2024-09-18T14:25:50.065831+0000","last_fullsized":"2024-09-18T14:25:50.065831+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4544,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:50.061953+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:50.061953+0000","last_peered":"2024-09-18T14:25:50.061953+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:50.061953+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":27097,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051607,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182424,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301816,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070024,"statfs":{"total":96636764160,"available":96327704576,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45600000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301892,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069948,"statfs":{"total":96636764160,"available":96327626752,"internally_reserved":0,"allocated":385024,"data_stored":181580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.629},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:44.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:45.097 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:45.112 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":635,"stamp":"2024-09-18T14:26:44.235126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":27097,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603708,"kb_used_data":680,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139972,"statfs":{"total":193273528320,"available":192655331328,"internally_reserved":0,"allocated":696320,"data_stored":307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2642,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002616"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:50.062116+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:50.062116+0000","last_peered":"2024-09-18T14:25:50.062116+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:50.062116+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4553,"reported_epoch":34,"state":"peering","last_fresh":"2024-09-18T14:26:42.314650+0000","last_change":"2024-09-18T14:26:42.314650+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:26:42.314650+0000","last_undegraded":"2024-09-18T14:26:42.314650+0000","last_fullsized":"2024-09-18T14:26:42.314650+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:39.400748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4673,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065831+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065831+0000","last_peered":"2024-09-18T14:25:50.065831+0000","last_clean":"2024-09-18T14:25:50.065831+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065831+0000","last_undegraded":"2024-09-18T14:25:50.065831+0000","last_fullsized":"2024-09-18T14:25:50.065831+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4544,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:50.061953+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:50.061953+0000","last_peered":"2024-09-18T14:25:50.061953+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:50.061953+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":27097,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051607,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182424,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301816,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070024,"statfs":{"total":96636764160,"available":96327704576,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45600000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301892,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069948,"statfs":{"total":96636764160,"available":96327626752,"internally_reserved":0,"allocated":385024,"data_stored":181580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.629},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:45.113 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:45.113 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4632, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061568+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:25:49.061568+0000', 'last_peered': '2024-09-18T14:25:49.061568+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:25:49.061568+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4602, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:50.062116+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:25:50.062116+0000', 'last_peered': '2024-09-18T14:25:50.062116+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:25:50.062116+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4498, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061277+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:25:49.061277+0000', 'last_peered': '2024-09-18T14:25:49.061277+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:25:49.061277+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:PG 1.4 is not active+clean 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.4', 'version': "17'4503", 'reported_seq': 4553, 'reported_epoch': 34, 'state': 'peering', 'last_fresh': '2024-09-18T14:26:42.314650+0000', 'last_change': '2024-09-18T14:26:42.314650+0000', 'last_active': '2024-09-18T14:24:00.775289+0000', 'last_peered': '2024-09-18T14:23:59.258639+0000', 'last_clean': '2024-09-18T14:23:59.258639+0000', 'last_became_active': '2024-09-18T14:06:51.051512+0000', 'last_became_peered': '2024-09-18T14:06:51.051512+0000', 'last_unstale': '2024-09-18T14:26:42.314650+0000', 'last_undegraded': '2024-09-18T14:26:42.314650+0000', 'last_fullsized': '2024-09-18T14:26:42.314650+0000', 'mapping_epoch': 34, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4503, 'log_dups_size': 0, 'ondisk_log_size': 4503, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T20:47:39.400748+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000658006, 'stat_sum': {'num_bytes': 56963, 'num_objects': 4503, 'num_object_clones': 0, 'num_object_copies': 9006, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 4503, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4503, 'num_write_kb': 4551, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [3], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 4530, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.061286+0000', 'last_change': '2024-09-18T14:24:48.420227+0000', 'last_active': '2024-09-18T14:25:49.061286+0000', 'last_peered': '2024-09-18T14:25:49.061286+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:25:49.061286+0000', 'last_undegraded': '2024-09-18T14:24:48.407099+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4482, 'num_objects_misplaced': 4482, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4482}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:45.114 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4544, 'reported_epoch': 33, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:50.061953+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:25:50.061953+0000', 'last_peered': '2024-09-18T14:25:50.061953+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:25:50.061953+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:45.115 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:45.115 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4567, 'reported_epoch': 32, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:25:49.062076+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:25:49.062076+0000', 'last_peered': '2024-09-18T14:25:49.062076+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:25:49.062076+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:45.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:45.481 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:45.481 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 1 pg inactive, 1 pg peering","count":2},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 27097/72472 objects degraded (37.390%), 6 pgs degraded, 6 pgs undersized","count":12},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1216,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":6},{"state_name":"active+clean","count":1},{"state_name":"peering","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618196992,"bytes_avail":192655331328,"bytes_total":193273528320,"inactive_pgs_ratio":0.125,"degraded_objects":27097,"degraded_total":72472,"degraded_ratio":0.37389612540015454,"misplaced_objects":27097,"misplaced_total":72472,"misplaced_ratio":0.37389612540015454},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:45.495 INFO:tasks.osd_recovery.ceph_manager:no progress seen, keeping timeout for now 2024-09-18T14:26:45.495 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:45.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:45.810 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:45.825 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":635,"stamp":"2024-09-18T14:26:44.235126+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":27097,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603708,"kb_used_data":680,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139972,"statfs":{"total":193273528320,"available":192655331328,"internally_reserved":0,"allocated":696320,"data_stored":307602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2642,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-2121,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002616"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4632,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061568+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:25:49.061568+0000","last_peered":"2024-09-18T14:25:49.061568+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:25:49.061568+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4602,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:50.062116+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:25:50.062116+0000","last_peered":"2024-09-18T14:25:50.062116+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:25:50.062116+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061277+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:25:49.061277+0000","last_peered":"2024-09-18T14:25:49.061277+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:25:49.061277+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4553,"reported_epoch":34,"state":"peering","last_fresh":"2024-09-18T14:26:42.314650+0000","last_change":"2024-09-18T14:26:42.314650+0000","last_active":"2024-09-18T14:24:00.775289+0000","last_peered":"2024-09-18T14:23:59.258639+0000","last_clean":"2024-09-18T14:23:59.258639+0000","last_became_active":"2024-09-18T14:06:51.051512+0000","last_became_peered":"2024-09-18T14:06:51.051512+0000","last_unstale":"2024-09-18T14:26:42.314650+0000","last_undegraded":"2024-09-18T14:26:42.314650+0000","last_fullsized":"2024-09-18T14:26:42.314650+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:47:39.400748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.061286+0000","last_change":"2024-09-18T14:24:48.420227+0000","last_active":"2024-09-18T14:25:49.061286+0000","last_peered":"2024-09-18T14:25:49.061286+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:25:49.061286+0000","last_undegraded":"2024-09-18T14:24:48.407099+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4482,"num_objects_misplaced":4482,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4482}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4673,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-09-18T14:25:50.065831+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:25:50.065831+0000","last_peered":"2024-09-18T14:25:50.065831+0000","last_clean":"2024-09-18T14:25:50.065831+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:25:50.065831+0000","last_undegraded":"2024-09-18T14:25:50.065831+0000","last_fullsized":"2024-09-18T14:25:50.065831+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4544,"reported_epoch":33,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:50.061953+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:25:50.061953+0000","last_peered":"2024-09-18T14:25:50.061953+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:25:50.061953+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":32,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:25:49.062076+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:25:49.062076+0000","last_peered":"2024-09-18T14:25:49.062076+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:25:49.062076+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":27097,"num_objects_misplaced":27097,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4638,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051607,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182424,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301816,"kb_used_data":304,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94070024,"statfs":{"total":96636764160,"available":96327704576,"internally_reserved":0,"allocated":311296,"data_stored":126022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.64600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.45600000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705913,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301892,"kb_used_data":376,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069948,"statfs":{"total":96636764160,"available":96327626752,"internally_reserved":0,"allocated":385024,"data_stored":181580,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.44600000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.629},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:48.827 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:49.135 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:49.138 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:49.148 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":637,"stamp":"2024-09-18T14:26:48.236363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":31119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8310,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":2512,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002556"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":5223,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:46.343708+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:46.343708+0000","last_peered":"2024-09-18T14:26:46.343708+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:46.343708+0000","last_undegraded":"2024-09-18T14:26:46.343708+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8504,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":230},{"shards":"2,3","objects":4252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":31119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8310,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051608,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705915,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.217},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.27900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:49.149 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:49.468 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:49.471 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:49.480 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":637,"stamp":"2024-09-18T14:26:48.236363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":31119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8310,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":2512,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002556"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":5223,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:46.343708+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:46.343708+0000","last_peered":"2024-09-18T14:26:46.343708+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:46.343708+0000","last_undegraded":"2024-09-18T14:26:46.343708+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8504,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":230},{"shards":"2,3","objects":4252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":31119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8310,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051608,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705915,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.217},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.27900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:49.481 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:49.481 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 5223, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:46.343708+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:26:46.343708+0000', 'last_peered': '2024-09-18T14:26:46.343708+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:26:46.343708+0000', 'last_undegraded': '2024-09-18T14:26:46.343708+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8504, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 231, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 230}, {'shards': '2,3', 'objects': 4252}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:49.482 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:49.483 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:49.483 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:49.831 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:49.831 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1220,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618283008,"bytes_avail":192655245312,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":31119,"misplaced_total":72472,"misplaced_ratio":0.42939342090738492,"recovering_objects_per_sec":123,"recovering_bytes_per_sec":6555,"recovering_keys_per_sec":0,"num_objects_recovered":985,"num_bytes_recovered":52464,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:49.845 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:49.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:50.151 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:50.154 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:50.165 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":637,"stamp":"2024-09-18T14:26:48.236363+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":31119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8310,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":2512,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":985,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002556"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":5223,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:46.343708+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:46.343708+0000","last_peered":"2024-09-18T14:26:46.343708+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:46.343708+0000","last_undegraded":"2024-09-18T14:26:46.343708+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8504,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":231,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":230},{"shards":"2,3","objects":4252}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":31119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8310,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051608,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182425,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705915,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.217},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:25:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":3.117,"5min":3.117,"15min":3.117},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":46.807000000000002,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.51700000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.27900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:53.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:53.492 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:53.495 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:53.507 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2024-09-18T14:26:52.237529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":1960,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.929942"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":6051,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:51.350491+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:51.350491+0000","last_peered":"2024-09-18T14:26:51.350491+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:51.350491+0000","last_undegraded":"2024-09-18T14:26:51.350491+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7952,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":506},{"shards":"2,3","objects":3976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051608,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.27900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:53.509 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:53.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:53.809 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:53.821 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":639,"stamp":"2024-09-18T14:26:52.237529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":1960,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1261,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.929942"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":6051,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:51.350491+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:51.350491+0000","last_peered":"2024-09-18T14:26:51.350491+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:51.350491+0000","last_undegraded":"2024-09-18T14:26:51.350491+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7952,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":506},{"shards":"2,3","objects":3976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051608,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705914,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.52800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:25:47 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.27900000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.40300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 6051, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:51.350491+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:26:51.350491+0000', 'last_peered': '2024-09-18T14:26:51.350491+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:26:51.350491+0000', 'last_undegraded': '2024-09-18T14:26:51.350491+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7952, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 507, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 506}, {'shards': '2,3', 'objects': 3976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:53.822 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:53.823 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:53.823 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:53.823 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:53.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:54.188 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:54.188 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1225,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618283008,"bytes_avail":192655245312,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":30567,"misplaced_total":72472,"misplaced_ratio":0.42177668616845126,"recovering_objects_per_sec":126,"recovering_bytes_per_sec":5283,"recovering_keys_per_sec":0,"num_objects_recovered":1261,"num_bytes_recovered":52464,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:54.205 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:54.205 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:54.501 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:54.503 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:54.514 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":640,"stamp":"2024-09-18T14:26:54.238303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":3470,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3948,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.922324"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":6051,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:51.350491+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:51.350491+0000","last_peered":"2024-09-18T14:26:51.350491+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:51.350491+0000","last_undegraded":"2024-09-18T14:26:51.350491+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7952,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":506},{"shards":"2,3","objects":3976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051609,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.85199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:57.516 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:57.815 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:57.818 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:57.828 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":641,"stamp":"2024-09-18T14:26:56.238839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":3470,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3948,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003713"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":6051,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:51.350491+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:51.350491+0000","last_peered":"2024-09-18T14:26:51.350491+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:51.350491+0000","last_undegraded":"2024-09-18T14:26:51.350491+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7952,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":506},{"shards":"2,3","objects":3976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051609,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.85199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:57.829 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:58.130 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:58.132 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:58.142 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":641,"stamp":"2024-09-18T14:26:56.238839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":3470,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3948,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003713"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":6051,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:51.350491+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:51.350491+0000","last_peered":"2024-09-18T14:26:51.350491+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:51.350491+0000","last_undegraded":"2024-09-18T14:26:51.350491+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7952,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":507,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":506},{"shards":"2,3","objects":3976}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":30567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8586,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051609,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182426,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59199999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.69499999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.56100000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705915,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.85199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.78900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:26:58.143 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:26:58.143 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:58.143 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:26:58.144 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:58.144 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:26:58.144 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:58.144 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:26:58.144 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 6051, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:26:51.350491+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:26:51.350491+0000', 'last_peered': '2024-09-18T14:26:51.350491+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:26:51.350491+0000', 'last_undegraded': '2024-09-18T14:26:51.350491+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7952, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 507, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 506}, {'shards': '2,3', 'objects': 3976}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:58.144 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:26:58.144 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:26:58.144 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:26:58.145 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:26:58.146 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:26:58.503 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:58.503 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1229,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626802688,"bytes_avail":192646725632,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":29991,"misplaced_total":72472,"misplaced_ratio":0.41382878904956399,"recovering_objects_per_sec":352,"recovering_bytes_per_sec":4370,"recovering_keys_per_sec":0,"num_objects_recovered":4236,"num_bytes_recovered":52464,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:26:58.518 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:26:58.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:26:58.811 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:26:58.815 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:26:58.830 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":642,"stamp":"2024-09-18T14:26:58.239532+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29991,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8874,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4482,"num_objects_misplaced":2894,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4236,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003717"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":6915,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:56.358639+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:56.358639+0000","last_peered":"2024-09-18T14:26:56.358639+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:56.358639+0000","last_undegraded":"2024-09-18T14:26:56.358639+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7376,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":794},{"shards":"2,3","objects":3688}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29991,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8874,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051610,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182427,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705917,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.83699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.68400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.53400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.39800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.499},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:01.832 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:02.131 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:02.134 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:02.144 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":643,"stamp":"2024-09-18T14:27:00.240029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29991,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8874,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1128,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":564,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003666"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":6915,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:26:56.358639+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:26:56.358639+0000","last_peered":"2024-09-18T14:26:56.358639+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:26:56.358639+0000","last_undegraded":"2024-09-18T14:26:56.358639+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7376,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":795,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":794},{"shards":"2,3","objects":3688}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29991,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8874,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051610,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182427,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705917,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.83699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.68400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.53400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.39800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.499},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:02.145 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:02.468 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:02.472 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:02.482 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2024-09-18T14:27:02.240609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1712,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003595"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":7792,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:01.355924+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:01.355924+0000","last_peered":"2024-09-18T14:27:01.355924+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:01.355924+0000","last_undegraded":"2024-09-18T14:27:01.355924+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6792,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1086},{"shards":"2,3","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051610,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705918,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.53400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.39800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.499},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:02.483 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:02.483 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:02.483 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:02.483 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:02.483 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:02.483 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:02.484 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:02.484 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 7792, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:01.355924+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:01.355924+0000', 'last_peered': '2024-09-18T14:27:01.355924+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:01.355924+0000', 'last_undegraded': '2024-09-18T14:27:01.355924+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6792, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1087, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1086}, {'shards': '2,3', 'objects': 3396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:02.484 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:02.484 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:02.484 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:02.484 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:02.484 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:02.851 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:02.851 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1233,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626802688,"bytes_avail":192646725632,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":29407,"misplaced_total":72472,"misplaced_ratio":0.40577050447069213,"recovering_objects_per_sec":71,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:02.865 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:02.866 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:03.202 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:03.217 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2024-09-18T14:27:02.240609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1712,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":856,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003595"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":7792,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:01.355924+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:01.355924+0000","last_peered":"2024-09-18T14:27:01.355924+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:01.355924+0000","last_undegraded":"2024-09-18T14:27:01.355924+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6792,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1086},{"shards":"2,3","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051610,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705918,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705916,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.53400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.39800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:25:57 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.499},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.47399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:06.218 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:06.536 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:06.539 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:06.550 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2024-09-18T14:27:06.241799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1160,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003495"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":7792,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:01.355924+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:01.355924+0000","last_peered":"2024-09-18T14:27:01.355924+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:01.355924+0000","last_undegraded":"2024-09-18T14:27:01.355924+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6792,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1086},{"shards":"2,3","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051611,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705918,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.626}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:06.551 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:06.883 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:06.886 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:06.896 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2024-09-18T14:27:06.241799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1160,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003495"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":7792,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:01.355924+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:01.355924+0000","last_peered":"2024-09-18T14:27:01.355924+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:01.355924+0000","last_undegraded":"2024-09-18T14:27:01.355924+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6792,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1086},{"shards":"2,3","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051611,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705918,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.626}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:06.897 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:06.897 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:06.897 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:06.897 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:06.897 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:06.897 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:06.898 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:06.898 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 7792, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:01.355924+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:01.355924+0000', 'last_peered': '2024-09-18T14:27:01.355924+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:01.355924+0000', 'last_undegraded': '2024-09-18T14:27:01.355924+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6792, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1087, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1086}, {'shards': '2,3', 'objects': 3396}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:06.898 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:06.898 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:06.898 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:06.898 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:06.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:07.264 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:07.264 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1238,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626802688,"bytes_avail":192646725632,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":29407,"misplaced_total":72472,"misplaced_ratio":0.40577050447069213,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:07.278 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:07.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:07.566 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:07.569 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:07.579 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":646,"stamp":"2024-09-18T14:27:06.241799+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1160,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":580,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003495"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":7792,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:01.355924+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:01.355924+0000","last_peered":"2024-09-18T14:27:01.355924+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:01.355924+0000","last_undegraded":"2024-09-18T14:27:01.355924+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6792,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1086},{"shards":"2,3","objects":3396}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":29407,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9166,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051611,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182428,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705918,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.76800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:26:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705917,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.626}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:10.581 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:10.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:10.892 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:10.904 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":648,"stamp":"2024-09-18T14:27:10.243278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28775,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9482,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1216,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003745"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":8739,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:06.358463+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:06.358463+0000","last_peered":"2024-09-18T14:27:06.358463+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:06.358463+0000","last_undegraded":"2024-09-18T14:27:06.358463+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6160,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1402},{"shards":"2,3","objects":3080}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28775,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9482,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051612,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182429,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705919,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.55000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.93400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.626}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:10.905 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:11.218 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:11.231 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":648,"stamp":"2024-09-18T14:27:10.243278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28775,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9482,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1216,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003745"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":8739,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:06.358463+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:06.358463+0000","last_peered":"2024-09-18T14:27:06.358463+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:06.358463+0000","last_undegraded":"2024-09-18T14:27:06.358463+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6160,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1402},{"shards":"2,3","objects":3080}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28775,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9482,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051612,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182429,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705919,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.55000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.93400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.626}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:11.232 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:11.232 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:11.232 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:11.232 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:11.232 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:11.232 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:11.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:11.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 8739, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:06.358463+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:06.358463+0000', 'last_peered': '2024-09-18T14:27:06.358463+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:06.358463+0000', 'last_undegraded': '2024-09-18T14:27:06.358463+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6160, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1403, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1402}, {'shards': '2,3', 'objects': 3080}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:11.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:11.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:11.233 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:11.233 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:11.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:11.594 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:11.594 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1242,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626802688,"bytes_avail":192646725632,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":28775,"misplaced_total":72472,"misplaced_ratio":0.39704989513191302,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":608,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:11.614 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:11.614 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:11.915 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:11.920 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:11.932 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":648,"stamp":"2024-09-18T14:27:10.243278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28775,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9482,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612112,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131568,"statfs":{"total":193273528320,"available":192646725632,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1216,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":608,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003745"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":8739,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:06.358463+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:06.358463+0000","last_peered":"2024-09-18T14:27:06.358463+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:06.358463+0000","last_undegraded":"2024-09-18T14:27:06.358463+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6160,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1403,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1402},{"shards":"2,3","objects":3080}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28775,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9482,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051612,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182429,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705919,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.55000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.93400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705918,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306056,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065784,"statfs":{"total":96636764160,"available":96323362816,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.626}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.38700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.65600000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:14.935 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:15.263 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:15.266 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:15.276 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":650,"stamp":"2024-09-18T14:27:14.244428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28109,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9815,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1298,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003819"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":9738,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:11.364309+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:11.364309+0000","last_peered":"2024-09-18T14:27:11.364309+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:11.364309+0000","last_undegraded":"2024-09-18T14:27:11.364309+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5494,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1735},{"shards":"2,3","objects":2747}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28109,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9815,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051613,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182430,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705920,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.441},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:15.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:15.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:15.611 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:15.621 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":650,"stamp":"2024-09-18T14:27:14.244428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28109,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9815,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1298,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003819"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":9738,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:11.364309+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:11.364309+0000","last_peered":"2024-09-18T14:27:11.364309+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:11.364309+0000","last_undegraded":"2024-09-18T14:27:11.364309+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5494,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1735},{"shards":"2,3","objects":2747}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28109,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9815,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051613,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182430,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705920,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.441},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:15.622 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:15.622 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 9738, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:11.364309+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:11.364309+0000', 'last_peered': '2024-09-18T14:27:11.364309+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:11.364309+0000', 'last_undegraded': '2024-09-18T14:27:11.364309+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5494, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1736, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1735}, {'shards': '2,3', 'objects': 2747}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:15.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:15.624 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:15.624 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:15.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:15.977 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1246,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635322368,"bytes_avail":192638205952,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":28109,"misplaced_total":72472,"misplaced_ratio":0.38786013908819961,"recovering_objects_per_sec":54,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":649,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:15.991 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:15.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:16.301 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:16.313 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":651,"stamp":"2024-09-18T14:27:16.244978+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28109,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9815,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1298,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":649,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003770"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":9738,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:11.364309+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:11.364309+0000","last_peered":"2024-09-18T14:27:11.364309+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:11.364309+0000","last_undegraded":"2024-09-18T14:27:11.364309+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5494,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1736,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1735},{"shards":"2,3","objects":2747}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":28109,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9815,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051613,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182430,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705920,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.54800000000000004,"15min":0.55500000000000005},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.67600000000000005,"5min":6.758,"15min":6.758},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.61799999999999999,"15min":0.60799999999999998},"min":{"1min":0.35199999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68400000000000005,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.441},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58599999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705919,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.46600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:19.315 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:19.642 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:19.644 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:19.655 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2024-09-18T14:27:18.245780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":27387,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10176,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2020,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1010,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003980"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":10821,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:16.365645+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:16.365645+0000","last_peered":"2024-09-18T14:27:16.365645+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:16.365645+0000","last_undegraded":"2024-09-18T14:27:16.365645+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4772,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2097,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2096},{"shards":"2,3","objects":2386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":27387,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10176,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051614,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.39700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.57899999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.98999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.93600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:19.657 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:19.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:19.981 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:19.992 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":652,"stamp":"2024-09-18T14:27:18.245780+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":27387,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10176,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2020,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1010,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003980"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":10821,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:16.365645+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:16.365645+0000","last_peered":"2024-09-18T14:27:16.365645+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:16.365645+0000","last_undegraded":"2024-09-18T14:27:16.365645+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4772,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2097,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2096},{"shards":"2,3","objects":2386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":27387,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10176,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051614,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.39700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.57899999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.98999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.93600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:19.993 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:19.993 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 10821, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:16.365645+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:16.365645+0000', 'last_peered': '2024-09-18T14:27:16.365645+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:16.365645+0000', 'last_undegraded': '2024-09-18T14:27:16.365645+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4772, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2097, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2096}, {'shards': '2,3', 'objects': 2386}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:19.994 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:19.995 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:19.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:20.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:20.372 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1251,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635322368,"bytes_avail":192638205952,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":27387,"misplaced_total":72472,"misplaced_ratio":0.37789767082459425,"recovering_objects_per_sec":84,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1010,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:20.389 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:20.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:20.695 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:27:20.688+0000 7f2012d9d700 0 --2- 172.21.15.16:0/4001681405 >> [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] conn(0x7f201412b780 0x7f201406d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:27:20.905 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:20.909 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:20.919 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":653,"stamp":"2024-09-18T14:27:20.246518+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":27387,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10176,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1388,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":694,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004037"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":10821,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:16.365645+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:16.365645+0000","last_peered":"2024-09-18T14:27:16.365645+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:16.365645+0000","last_undegraded":"2024-09-18T14:27:16.365645+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4772,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2097,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2096},{"shards":"2,3","objects":2386}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":27387,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10176,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051614,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182431,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.39700000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45500000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.57899999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.98999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.93600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:23.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:24.254 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:24.257 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:24.268 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":654,"stamp":"2024-09-18T14:27:22.247302+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":26625,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10557,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2150,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1075,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004024"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":11964,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:21.368140+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:21.368140+0000","last_peered":"2024-09-18T14:27:21.368140+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:21.368140+0000","last_undegraded":"2024-09-18T14:27:21.368140+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4010,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2477},{"shards":"2,3","objects":2005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":26625,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10557,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051614,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182432,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.438},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.34899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58899999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705920,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.51700000000000002,"15min":0.54500000000000004},"min":{"1min":0.27200000000000002,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.59999999999999998,"15min":0.66500000000000004},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.84399999999999997,"15min":0.90700000000000003},"last":0.98999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.93600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:24.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:24.561 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:24.565 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:24.577 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2024-09-18T14:27:24.251456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":26625,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10557,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1484,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007589"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":11964,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:21.368140+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:21.368140+0000","last_peered":"2024-09-18T14:27:21.368140+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:21.368140+0000","last_undegraded":"2024-09-18T14:27:21.368140+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4010,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2477},{"shards":"2,3","objects":2005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":26625,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10557,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051615,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182432,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.438},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.34899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58899999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:24.578 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:24.578 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:24.578 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:24.578 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:24.579 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:24.579 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:24.579 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:24.579 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 11964, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:21.368140+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:21.368140+0000', 'last_peered': '2024-09-18T14:27:21.368140+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:21.368140+0000', 'last_undegraded': '2024-09-18T14:27:21.368140+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4010, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2478, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2477}, {'shards': '2,3', 'objects': 2005}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:24.579 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:24.579 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:24.579 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:24.579 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:24.579 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:24.951 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:24.951 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1255,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635322368,"bytes_avail":192638205952,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":26625,"misplaced_total":72472,"misplaced_ratio":0.36738326526106635,"recovering_objects_per_sec":61,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":742,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:24.964 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:24.965 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:25.282 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:25.286 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:25.297 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":655,"stamp":"2024-09-18T14:27:24.251456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":26625,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10557,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620432,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123248,"statfs":{"total":193273528320,"available":192638205952,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1484,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":742,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007589"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":11964,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:21.368140+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:21.368140+0000","last_peered":"2024-09-18T14:27:21.368140+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:21.368140+0000","last_undegraded":"2024-09-18T14:27:21.368140+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4010,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2478,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2477},{"shards":"2,3","objects":2005}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":26625,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10557,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051615,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182432,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.438},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.34899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.58899999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705921,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310216,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061624,"statfs":{"total":96636764160,"available":96319102976,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:28.299 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:28.609 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:28.613 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:28.625 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2024-09-18T14:27:28.252728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":25763,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10988,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595472,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148208,"statfs":{"total":193273528320,"available":192663764992,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2346,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007749"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":13257,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:26.372888+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:26.372888+0000","last_peered":"2024-09-18T14:27:26.372888+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:26.372888+0000","last_undegraded":"2024-09-18T14:27:26.372888+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3148,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2908},{"shards":"2,3","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":25763,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10988,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705923,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.504}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58099999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:28.626 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:28.942 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:28.945 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:28.955 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2024-09-18T14:27:28.252728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":25763,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10988,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595472,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148208,"statfs":{"total":193273528320,"available":192663764992,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2346,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007749"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":13257,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:26.372888+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:26.372888+0000","last_peered":"2024-09-18T14:27:26.372888+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:26.372888+0000","last_undegraded":"2024-09-18T14:27:26.372888+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3148,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2908},{"shards":"2,3","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":25763,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10988,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705923,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.504}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58099999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:28.956 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:28.956 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:28.956 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:28.956 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:28.956 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:28.956 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:28.956 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:28.956 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 13257, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:26.372888+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:26.372888+0000', 'last_peered': '2024-09-18T14:27:26.372888+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:26.372888+0000', 'last_undegraded': '2024-09-18T14:27:26.372888+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3148, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2909, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2908}, {'shards': '2,3', 'objects': 1574}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:28.957 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:28.957 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:28.957 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:28.957 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:28.957 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:29.350 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:29.350 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1260,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609763328,"bytes_avail":192663764992,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":25763,"misplaced_total":72472,"misplaced_ratio":0.35548901644773151,"recovering_objects_per_sec":97,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1173,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:29.365 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:29.366 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:29.683 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:29.685 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:29.696 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":657,"stamp":"2024-09-18T14:27:28.252728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":25763,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10988,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595472,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148208,"statfs":{"total":193273528320,"available":192663764992,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2346,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1173,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007749"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":13257,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:26.372888+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:26.372888+0000","last_peered":"2024-09-18T14:27:26.372888+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:26.372888+0000","last_undegraded":"2024-09-18T14:27:26.372888+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3148,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2909,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2908},{"shards":"2,3","objects":1574}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":25763,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10988,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182433,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705923,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.504}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58099999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.61899999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:32.699 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:33.017 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:33.020 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:33.032 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-09-18T14:27:32.253935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595472,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148208,"statfs":{"total":193273528320,"available":192663764992,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2622,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007417"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":14754,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:31.371158+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:31.371158+0000","last_peered":"2024-09-18T14:27:31.371158+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:31.371158+0000","last_undegraded":"2024-09-18T14:27:31.371158+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3407},{"shards":"2,3","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.627},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71999999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:33.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:33.331 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:33.346 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-09-18T14:27:32.253935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595472,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148208,"statfs":{"total":193273528320,"available":192663764992,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2622,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007417"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":14754,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:31.371158+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:31.371158+0000","last_peered":"2024-09-18T14:27:31.371158+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:31.371158+0000","last_undegraded":"2024-09-18T14:27:31.371158+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3407},{"shards":"2,3","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.627},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71999999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:33.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:33.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:33.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:33.347 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:33.347 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:33.348 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:33.348 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:33.348 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 14754, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:31.371158+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:31.371158+0000', 'last_peered': '2024-09-18T14:27:31.371158+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:31.371158+0000', 'last_undegraded': '2024-09-18T14:27:31.371158+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2150, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3408, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3407}, {'shards': '2,3', 'objects': 1075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:33.348 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:33.348 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:33.348 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:33.348 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:33.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:33.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:33.809 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1264,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609763328,"bytes_avail":192663764992,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":24765,"misplaced_total":72472,"misplaced_ratio":0.34171818081465943,"recovering_objects_per_sec":109,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:33.823 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:33.823 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:34.136 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:34.139 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:34.153 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":659,"stamp":"2024-09-18T14:27:32.253935+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595472,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148208,"statfs":{"total":193273528320,"available":192663764992,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2622,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1311,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.007417"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":14754,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:31.371158+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:31.371158+0000","last_peered":"2024-09-18T14:27:31.371158+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:31.371158+0000","last_undegraded":"2024-09-18T14:27:31.371158+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3407},{"shards":"2,3","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051616,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.627},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71999999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705922,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.76500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:37.155 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:37.498 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:37.501 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:37.512 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2024-09-18T14:27:36.255246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595472,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148208,"statfs":{"total":193273528320,"available":192663764992,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1860,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003789"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":14754,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:31.371158+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:31.371158+0000","last_peered":"2024-09-18T14:27:31.371158+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:31.371158+0000","last_undegraded":"2024-09-18T14:27:31.371158+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3407},{"shards":"2,3","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051617,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.627},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71999999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.442},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:37.513 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:37.836 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:37.850 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":661,"stamp":"2024-09-18T14:27:36.255246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595472,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148208,"statfs":{"total":193273528320,"available":192663764992,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1860,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003789"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":14754,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:31.371158+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:31.371158+0000","last_peered":"2024-09-18T14:27:31.371158+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:31.371158+0000","last_undegraded":"2024-09-18T14:27:31.371158+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2150,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3407},{"shards":"2,3","objects":1075}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":24765,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11487,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051617,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182434,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.59699999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.627},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71999999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705923,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297736,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074104,"statfs":{"total":96636764160,"available":96331882496,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.48199999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.442},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.45500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:37.850 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 14754, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:31.371158+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:31.371158+0000', 'last_peered': '2024-09-18T14:27:31.371158+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:31.371158+0000', 'last_undegraded': '2024-09-18T14:27:31.371158+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2150, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3408, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3407}, {'shards': '2,3', 'objects': 1075}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:37.851 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:37.852 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:37.852 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:37.852 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:38.254 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:38.254 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 6 pgs undersized","count":11},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1269,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":35,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":6},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609763328,"bytes_avail":192663764992,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":24765,"misplaced_total":72472,"misplaced_ratio":0.34171818081465943,"recovering_objects_per_sec":77,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":930,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:38.269 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:38.269 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:38.586 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:38.589 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:38.604 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":663,"stamp":"2024-09-18T14:27:38.534763+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":23465,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12137,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2298,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1149,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.282035"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":16704,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:36.375289+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:36.375289+0000","last_peered":"2024-09-18T14:27:36.375289+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:36.375289+0000","last_undegraded":"2024-09-18T14:27:36.375289+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":850,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4058,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4057},{"shards":"2,3","objects":425}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":23465,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12137,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051618,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182435,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705925,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.42699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.36599999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.375},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.28399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:41.606 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:41.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:41.902 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:41.912 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2024-09-18T14:27:40.256224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":23465,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12137,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1300,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002288"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":16704,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:36.375289+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:36.375289+0000","last_peered":"2024-09-18T14:27:36.375289+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:36.375289+0000","last_undegraded":"2024-09-18T14:27:36.375289+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":850,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4058,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4057},{"shards":"2,3","objects":425}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":23465,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12137,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051618,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182435,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705925,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.42699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.36599999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.375},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.28399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:41.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:42.224 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:42.227 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:42.239 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":665,"stamp":"2024-09-18T14:27:40.256224+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":23465,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12137,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603792,"kb_used_data":768,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139888,"statfs":{"total":193273528320,"available":192655245312,"internally_reserved":0,"allocated":786432,"data_stored":370083,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1300,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":650,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002288"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4634,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313349+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:26:42.313349+0000","last_peered":"2024-09-18T14:26:42.313349+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:26:42.313349+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4500,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313610+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:26:42.313610+0000","last_peered":"2024-09-18T14:26:42.313610+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:26:42.313610+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":16704,"reported_epoch":35,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:36.375289+0000","last_change":"2024-09-18T14:26:41.981319+0000","last_active":"2024-09-18T14:27:36.375289+0000","last_peered":"2024-09-18T14:27:36.375289+0000","last_clean":"2024-09-18T14:23:58.621906+0000","last_became_active":"2024-09-18T14:24:48.413855+0000","last_became_peered":"2024-09-18T14:24:48.413855+0000","last_unstale":"2024-09-18T14:27:36.375289+0000","last_undegraded":"2024-09-18T14:27:36.375289+0000","last_fullsized":"2024-09-18T14:24:48.308989+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":850,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4058,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4057},{"shards":"2,3","objects":425}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.313188+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:26:42.313188+0000","last_peered":"2024-09-18T14:26:42.313188+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:26:42.313188+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":23465,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12137,"num_bytes_recovered":78734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":843776,"data_stored":655924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":10,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051618,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182435,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705925,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":185171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.42699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.36599999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.68999999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.375},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.28399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:42.239 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4634, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313349+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:26:42.313349+0000', 'last_peered': '2024-09-18T14:26:42.313349+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:26:42.313349+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4603, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.319804+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:26:42.319804+0000', 'last_peered': '2024-09-18T14:26:42.319804+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:26:42.319804+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4500, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313610+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:26:42.313610+0000', 'last_peered': '2024-09-18T14:26:42.313610+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:26:42.313610+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:PG 1.3 is not active+clean 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.3', 'version': "17'4482", 'reported_seq': 16704, 'reported_epoch': 35, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:36.375289+0000', 'last_change': '2024-09-18T14:26:41.981319+0000', 'last_active': '2024-09-18T14:27:36.375289+0000', 'last_peered': '2024-09-18T14:27:36.375289+0000', 'last_clean': '2024-09-18T14:23:58.621906+0000', 'last_became_active': '2024-09-18T14:24:48.413855+0000', 'last_became_peered': '2024-09-18T14:24:48.413855+0000', 'last_unstale': '2024-09-18T14:27:36.375289+0000', 'last_undegraded': '2024-09-18T14:27:36.375289+0000', 'last_fullsized': '2024-09-18T14:24:48.308989+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4482, 'log_dups_size': 0, 'ondisk_log_size': 4482, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T23:44:37.412965+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 30711, 'num_objects': 4482, 'num_object_clones': 0, 'num_object_copies': 8964, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 850, 'num_objects_unfound': 0, 'num_objects_dirty': 4482, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4481, 'num_write_kb': 4505, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4058, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 4057}, {'shards': '2,3', 'objects': 425}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:42.240 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 4545, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.320121+0000', 'last_change': '2024-09-18T14:24:59.567927+0000', 'last_active': '2024-09-18T14:26:42.320121+0000', 'last_peered': '2024-09-18T14:26:42.320121+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:26:42.320121+0000', 'last_undegraded': '2024-09-18T14:24:59.550821+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4490, 'num_objects_misplaced': 4490, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4490}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:42.241 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:42.241 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4569, 'reported_epoch': 34, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:26:42.313188+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:26:42.313188+0000', 'last_peered': '2024-09-18T14:26:42.313188+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:26:42.313188+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:42.241 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:42.623 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:42.623 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 22615/72472 objects degraded (31.205%), 5 pgs degraded, 5 pgs undersized","count":10},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1273,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":5},{"state_name":"active+clean","count":3}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618332160,"bytes_avail":192655196160,"bytes_total":193273528320,"degraded_objects":22615,"degraded_total":72472,"degraded_ratio":0.31205155094381276,"misplaced_objects":22615,"misplaced_total":72472,"misplaced_ratio":0.31205155094381276,"recovering_objects_per_sec":134,"recovering_bytes_per_sec":3278,"recovering_keys_per_sec":0,"num_objects_recovered":1075,"num_bytes_recovered":26232,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:42.637 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:42.637 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:42.934 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:42.937 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:42.949 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":666,"stamp":"2024-09-18T14:27:42.256704+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":22615,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12562,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603840,"kb_used_data":816,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139840,"statfs":{"total":193273528320,"available":192655196160,"internally_reserved":0,"allocated":835584,"data_stored":402950,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2150,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1075,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002036"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4603,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.319804+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:26:42.319804+0000","last_peered":"2024-09-18T14:26:42.319804+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:26:42.319804+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4561,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-18T14:26:43.338226+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:26:43.338226+0000","last_peered":"2024-09-18T14:26:43.338226+0000","last_clean":"2024-09-18T14:26:43.338226+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:26:43.338226+0000","last_undegraded":"2024-09-18T14:26:43.338226+0000","last_fullsized":"2024-09-18T14:26:43.338226+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4674,"reported_epoch":34,"state":"active+clean","last_fresh":"2024-09-18T14:26:42.313345+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:26:42.313345+0000","last_peered":"2024-09-18T14:26:42.313345+0000","last_clean":"2024-09-18T14:26:42.313345+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:26:42.313345+0000","last_undegraded":"2024-09-18T14:26:42.313345+0000","last_fullsized":"2024-09-18T14:26:42.313345+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4545,"reported_epoch":34,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:26:42.320121+0000","last_change":"2024-09-18T14:24:59.567927+0000","last_active":"2024-09-18T14:26:42.320121+0000","last_peered":"2024-09-18T14:26:42.320121+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:26:42.320121+0000","last_undegraded":"2024-09-18T14:24:59.550821+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4490,"num_objects_misplaced":4490,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4490}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22615,"num_objects_misplaced":22615,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12562,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":876544,"data_stored":682156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051618,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.309},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.41399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705924,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301896,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069944,"statfs":{"total":96636764160,"available":96327622656,"internally_reserved":0,"allocated":393216,"data_stored":184912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.375},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.42999999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.28399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":78715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:45.950 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:46.263 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:46.266 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:46.277 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-09-18T14:27:46.257759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26701,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12764,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603888,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139792,"statfs":{"total":193273528320,"available":192655147008,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":3236,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002032"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":5154,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:42.455734+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:42.455734+0000","last_peered":"2024-09-18T14:27:42.455734+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:42.455734+0000","last_undegraded":"2024-09-18T14:27:42.455734+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8576,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4288},{"shards":"3","objects":202}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26701,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12764,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051619,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.309},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.41399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.379}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:46.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:46.585 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:46.587 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:46.598 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-09-18T14:27:46.257759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26701,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12764,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603888,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139792,"statfs":{"total":193273528320,"available":192655147008,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":3236,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002032"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":5154,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:42.455734+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:42.455734+0000","last_peered":"2024-09-18T14:27:42.455734+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:42.455734+0000","last_undegraded":"2024-09-18T14:27:42.455734+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8576,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4288},{"shards":"3","objects":202}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26701,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12764,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051619,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.309},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.41399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.379}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 5154, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:42.455734+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:27:42.455734+0000', 'last_peered': '2024-09-18T14:27:42.455734+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:27:42.455734+0000', 'last_undegraded': '2024-09-18T14:27:42.455734+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8576, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 202, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4288}, {'shards': '3', 'objects': 202}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:46.599 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:46.600 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:46.600 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:47.019 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:47.019 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1277,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618381312,"bytes_avail":192655147008,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":26701,"misplaced_total":72472,"misplaced_ratio":0.36843194613091951,"recovering_objects_per_sec":78,"recovering_bytes_per_sec":3278,"recovering_keys_per_sec":0,"num_objects_recovered":627,"num_bytes_recovered":26232,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:47.035 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:47.036 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:47.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:47.356 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:47.366 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":668,"stamp":"2024-09-18T14:27:46.257759+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26701,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12764,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603888,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139792,"statfs":{"total":193273528320,"available":192655147008,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":3236,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002032"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":5154,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:42.455734+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:42.455734+0000","last_peered":"2024-09-18T14:27:42.455734+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:42.455734+0000","last_undegraded":"2024-09-18T14:27:42.455734+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8576,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4288},{"shards":"3","objects":202}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26701,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12764,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051619,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.309},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.70699999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.41399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705925,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.26000000000000001},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.379}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.41599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.36099999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:50.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:50.667 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:50.670 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:50.686 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2024-09-18T14:27:50.258729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26158,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13035,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":2693,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":898,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.716263"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":5969,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:47.473216+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:47.473216+0000","last_peered":"2024-09-18T14:27:47.473216+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:47.473216+0000","last_undegraded":"2024-09-18T14:27:47.473216+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8033,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"1,2,3","objects":1},{"shards":"2,3","objects":4016},{"shards":"3","objects":473}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26158,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13035,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051620,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705927,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.52000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.498},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.48199999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.95299999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.40600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.438}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:50.687 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:51.003 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:51.006 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:51.016 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2024-09-18T14:27:50.258729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26158,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13035,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":2693,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":898,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.716263"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":5969,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:47.473216+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:47.473216+0000","last_peered":"2024-09-18T14:27:47.473216+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:47.473216+0000","last_undegraded":"2024-09-18T14:27:47.473216+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8033,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"1,2,3","objects":1},{"shards":"2,3","objects":4016},{"shards":"3","objects":473}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26158,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13035,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051620,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705927,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.52000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.498},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.48199999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.95299999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.40600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.438}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:51.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:51.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:51.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:51.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:51.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:51.017 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:51.017 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:51.018 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 5969, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:47.473216+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:27:47.473216+0000', 'last_peered': '2024-09-18T14:27:47.473216+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:27:47.473216+0000', 'last_undegraded': '2024-09-18T14:27:47.473216+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8033, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 473, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '1,2,3', 'objects': 1}, {'shards': '2,3', 'objects': 4016}, {'shards': '3', 'objects': 473}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:51.018 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:51.018 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:51.018 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:51.420 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:51.421 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1282,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626900992,"bytes_avail":192646627328,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":26158,"misplaced_total":72472,"misplaced_ratio":0.3609393972844685,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":2447,"recovering_keys_per_sec":0,"num_objects_recovered":898,"num_bytes_recovered":26232,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:51.436 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:51.436 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:51.738 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:51.742 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:51.753 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":670,"stamp":"2024-09-18T14:27:50.258729+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26158,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13035,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":2693,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":898,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.716263"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":5969,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:47.473216+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:47.473216+0000","last_peered":"2024-09-18T14:27:47.473216+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:47.473216+0000","last_undegraded":"2024-09-18T14:27:47.473216+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8033,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":473,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"1,2,3","objects":1},{"shards":"2,3","objects":4016},{"shards":"3","objects":473}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":26158,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13035,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051620,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182437,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705927,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.52000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:26:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.47999999999999998,"15min":0.47399999999999998},"min":{"1min":0.33100000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.69199999999999995,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.498},{"interface":"front","average":{"1min":0.58699999999999997,"5min":2.6110000000000002,"15min":2.948},"min":{"1min":0.31,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.93100000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.48199999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.95299999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705926,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.40600000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:26:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.50900000000000001,"15min":0.495},"min":{"1min":0.42999999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.27900000000000003,"5min":0.247,"15min":0.247},"max":{"1min":3.1920000000000002,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.438}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.48899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:54.755 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:55.068 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:55.070 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:55.080 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2024-09-18T14:27:54.259959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25597,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13316,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":2982,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":6810,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:52.467811+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:52.467811+0000","last_peered":"2024-09-18T14:27:52.467811+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:52.467811+0000","last_undegraded":"2024-09-18T14:27:52.467811+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7472,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3736},{"shards":"3","objects":754}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25597,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13316,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051621,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182438,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705928,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":1.1339999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:55.082 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:55.368 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:55.372 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:55.386 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2024-09-18T14:27:54.259959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25597,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13316,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":2982,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":6810,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:52.467811+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:52.467811+0000","last_peered":"2024-09-18T14:27:52.467811+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:52.467811+0000","last_undegraded":"2024-09-18T14:27:52.467811+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7472,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3736},{"shards":"3","objects":754}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25597,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13316,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051621,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182438,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705928,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":1.1339999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:55.387 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:55.387 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:55.387 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:55.387 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:55.387 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:55.387 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:55.387 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:55.388 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 6810, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:52.467811+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:27:52.467811+0000', 'last_peered': '2024-09-18T14:27:52.467811+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:27:52.467811+0000', 'last_undegraded': '2024-09-18T14:27:52.467811+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7472, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 754, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3736}, {'shards': '3', 'objects': 754}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:55.388 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:55.388 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:55.388 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:27:55.756 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:55.756 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1286,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626900992,"bytes_avail":192646627328,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":25597,"misplaced_total":72472,"misplaced_ratio":0.35319847665305221,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:27:55.769 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:27:55.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:56.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:56.061 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:56.072 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":672,"stamp":"2024-09-18T14:27:54.259959+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25597,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13316,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4490,"num_objects_misplaced":2982,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":6810,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:52.467811+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:52.467811+0000","last_peered":"2024-09-18T14:27:52.467811+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:52.467811+0000","last_undegraded":"2024-09-18T14:27:52.467811+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7472,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3736},{"shards":"3","objects":754}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25597,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13316,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051621,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182438,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705928,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.432}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":1.1339999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705927,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.59899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.52100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:59.074 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:59.386 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:59.390 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:59.402 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2024-09-18T14:27:58.261043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25013,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13608,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1688,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":7686,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:57.476292+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:57.476292+0000","last_peered":"2024-09-18T14:27:57.476292+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:57.476292+0000","last_undegraded":"2024-09-18T14:27:57.476292+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6888,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3444},{"shards":"3","objects":1046}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25013,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13608,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051622,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705929,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.626}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.78300000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.37}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:59.403 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:27:59.708 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:27:59.711 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:27:59.722 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":674,"stamp":"2024-09-18T14:27:58.261043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25013,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13608,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1688,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003283"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":7686,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:57.476292+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:57.476292+0000","last_peered":"2024-09-18T14:27:57.476292+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:57.476292+0000","last_undegraded":"2024-09-18T14:27:57.476292+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6888,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3444},{"shards":"3","objects":1046}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25013,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13608,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051622,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705929,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.626}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.78300000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.37}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:27:59.723 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:27:59.723 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:59.724 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:27:59.724 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:59.724 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:27:59.724 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:59.724 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:27:59.724 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 7686, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:27:57.476292+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:27:57.476292+0000', 'last_peered': '2024-09-18T14:27:57.476292+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:27:57.476292+0000', 'last_undegraded': '2024-09-18T14:27:57.476292+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6888, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1046, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3444}, {'shards': '3', 'objects': 1046}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:27:59.724 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:27:59.724 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:27:59.724 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:00.095 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:00.095 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1290,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626900992,"bytes_avail":192646627328,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":25013,"misplaced_total":72472,"misplaced_ratio":0.3451401920741804,"recovering_objects_per_sec":70,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:00.110 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:00.110 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:00.437 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:00.440 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:00.453 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":675,"stamp":"2024-09-18T14:28:00.261456+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25013,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13608,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003340"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":7686,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:57.476292+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:57.476292+0000","last_peered":"2024-09-18T14:27:57.476292+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:57.476292+0000","last_undegraded":"2024-09-18T14:27:57.476292+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6888,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3444},{"shards":"3","objects":1046}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25013,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13608,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051622,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182439,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705929,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.67500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.626}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.78300000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.37}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:03.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:03.791 INFO:teuthology.orchestra.run.smithi016.stderr:2024-09-18T14:28:03.787+0000 7fb7f4945700 0 --2- 172.21.15.16:0/582564768 >> [v2:172.21.15.16:3301/0,v1:172.21.15.16:6790/0] conn(0x7fb7f008ec70 0x7fb7f008b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-18T14:28:03.985 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:03.989 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:04.000 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":676,"stamp":"2024-09-18T14:28:02.262032+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25013,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13608,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616368,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188127312,"statfs":{"total":193273528320,"available":192642367488,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1145,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":573,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003303"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":7686,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:27:57.476292+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:27:57.476292+0000","last_peered":"2024-09-18T14:27:57.476292+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:27:57.476292+0000","last_undegraded":"2024-09-18T14:27:57.476292+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6888,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1046,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3444},{"shards":"3","objects":1046}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":25013,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13608,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051622,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182440,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.74099999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":1.107}]}]},{"osd":0,"up_from":9,"seq":38654705928,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306104,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065736,"statfs":{"total":96636764160,"available":96323313664,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.37}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.46200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:04.001 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:04.324 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:04.327 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:04.344 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2024-09-18T14:28:04.262659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":24415,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13907,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1743,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003228"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":8583,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:02.477297+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:02.477297+0000","last_peered":"2024-09-18T14:28:02.477297+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:02.477297+0000","last_undegraded":"2024-09-18T14:28:02.477297+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6290,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1345,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3145},{"shards":"3","objects":1345}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":24415,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13907,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051623,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182440,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.74099999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":1.107}]}]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.88200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.753},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.70699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:04.345 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:04.345 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:04.345 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:04.346 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:04.346 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:04.346 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:04.346 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:28:04.346 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 8583, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:02.477297+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:28:02.477297+0000', 'last_peered': '2024-09-18T14:28:02.477297+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:28:02.477297+0000', 'last_undegraded': '2024-09-18T14:28:02.477297+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6290, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1345, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3145}, {'shards': '3', 'objects': 1345}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:04.346 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:04.346 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:04.346 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:04.701 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:04.701 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1295,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635420672,"bytes_avail":192638107648,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":24415,"misplaced_total":72472,"misplaced_ratio":0.3368887294403356,"recovering_objects_per_sec":72,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":872,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:04.716 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:04.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:05.017 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:05.020 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:05.031 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":677,"stamp":"2024-09-18T14:28:04.262659+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":24415,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13907,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1743,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":872,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003228"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":8583,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:02.477297+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:02.477297+0000","last_peered":"2024-09-18T14:28:02.477297+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:02.477297+0000","last_undegraded":"2024-09-18T14:28:02.477297+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6290,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1345,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3145},{"shards":"3","objects":1345}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":24415,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13907,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051623,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182440,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.74099999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.66700000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:27:02 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.54400000000000004,"15min":0.53500000000000003},"min":{"1min":0.34300000000000003,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.99199999999999999},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.627,"15min":0.621},"min":{"1min":0.45600000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":1.107}]}]},{"osd":0,"up_from":9,"seq":38654705929,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.88200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:26:59 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.625,"15min":0.61799999999999999},"min":{"1min":0.38500000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.753},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.621,"15min":0.61899999999999999},"min":{"1min":0.40300000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.70699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:08.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:08.342 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:08.358 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2024-09-18T14:28:08.263813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1818,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003213"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:07.480891+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:07.480891+0000","last_peered":"2024-09-18T14:28:07.480891+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:07.480891+0000","last_undegraded":"2024-09-18T14:28:07.480891+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5654,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1663,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2827},{"shards":"3","objects":1663}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705931,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.374},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.44600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.624}]}]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:08.359 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:08.646 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:08.649 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:08.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2024-09-18T14:28:08.263813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1818,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003213"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:07.480891+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:07.480891+0000","last_peered":"2024-09-18T14:28:07.480891+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:07.480891+0000","last_undegraded":"2024-09-18T14:28:07.480891+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5654,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1663,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2827},{"shards":"3","objects":1663}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705931,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.374},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.44600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.624}]}]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 9537, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:07.480891+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:28:07.480891+0000', 'last_peered': '2024-09-18T14:28:07.480891+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:28:07.480891+0000', 'last_undegraded': '2024-09-18T14:28:07.480891+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5654, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1663, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2827}, {'shards': '3', 'objects': 1663}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:08.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:08.661 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:08.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:09.014 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:09.014 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1299,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635420672,"bytes_avail":192638107648,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":23779,"misplaced_total":72472,"misplaced_ratio":0.3281129263715642,"recovering_objects_per_sec":75,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":909,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:09.032 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:09.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:09.332 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:09.347 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":679,"stamp":"2024-09-18T14:28:08.263813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1818,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":909,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003213"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:07.480891+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:07.480891+0000","last_peered":"2024-09-18T14:28:07.480891+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:07.480891+0000","last_undegraded":"2024-09-18T14:28:07.480891+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5654,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1663,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2827},{"shards":"3","objects":1663}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705931,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.54900000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.374},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.44600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.624}]}]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:12.348 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:12.649 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:12.652 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:12.667 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2024-09-18T14:28:12.264761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1234,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003305"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:07.480891+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:07.480891+0000","last_peered":"2024-09-18T14:28:07.480891+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:07.480891+0000","last_undegraded":"2024-09-18T14:28:07.480891+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5654,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1663,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2827},{"shards":"3","objects":1663}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.628},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.96399999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:12.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:12.967 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:12.980 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2024-09-18T14:28:12.264761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1234,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003305"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:07.480891+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:07.480891+0000","last_peered":"2024-09-18T14:28:07.480891+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:07.480891+0000","last_undegraded":"2024-09-18T14:28:07.480891+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5654,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1663,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2827},{"shards":"3","objects":1663}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.628},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.96399999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:12.980 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 9537, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:07.480891+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:28:07.480891+0000', 'last_peered': '2024-09-18T14:28:07.480891+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:28:07.480891+0000', 'last_undegraded': '2024-09-18T14:28:07.480891+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5654, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1663, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2827}, {'shards': '3', 'objects': 1663}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:12.981 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:12.982 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:13.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:13.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1304,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635420672,"bytes_avail":192638107648,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":23779,"misplaced_total":72472,"misplaced_ratio":0.3281129263715642,"recovering_objects_per_sec":51,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":617,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:13.367 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:13.367 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:13.706 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:13.709 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:13.720 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":681,"stamp":"2024-09-18T14:28:12.264761+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1234,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":617,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003305"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":9537,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:07.480891+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:07.480891+0000","last_peered":"2024-09-18T14:28:07.480891+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:07.480891+0000","last_undegraded":"2024-09-18T14:28:07.480891+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5654,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1663,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2827},{"shards":"3","objects":1663}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23779,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14225,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.628},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.96399999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705930,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.72099999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44400000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.46500000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:16.722 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:17.027 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:17.032 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:17.041 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2024-09-18T14:28:16.265913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23083,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14573,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1332,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":666,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":10581,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:12.483106+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:12.483106+0000","last_peered":"2024-09-18T14:28:12.483106+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:12.483106+0000","last_undegraded":"2024-09-18T14:28:12.483106+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4958,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2479},{"shards":"3","objects":2011}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23083,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14573,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051625,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.628},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.96399999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:17.042 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:17.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:17.365 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:17.376 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2024-09-18T14:28:16.265913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23083,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14573,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1332,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":666,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":10581,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:12.483106+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:12.483106+0000","last_peered":"2024-09-18T14:28:12.483106+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:12.483106+0000","last_undegraded":"2024-09-18T14:28:12.483106+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4958,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2479},{"shards":"3","objects":2011}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23083,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14573,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051625,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.628},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.96399999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:17.377 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:17.377 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:17.378 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:17.378 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:17.378 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:17.378 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:17.378 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:28:17.378 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 10581, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:12.483106+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:28:12.483106+0000', 'last_peered': '2024-09-18T14:28:12.483106+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:28:12.483106+0000', 'last_undegraded': '2024-09-18T14:28:12.483106+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4958, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2011, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2479}, {'shards': '3', 'objects': 2011}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:17.378 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:17.378 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:17.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:17.755 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:17.755 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1308,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635420672,"bytes_avail":192638107648,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":23083,"misplaced_total":72472,"misplaced_ratio":0.31850921735290871,"recovering_objects_per_sec":55,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":666,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:17.771 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:17.771 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:18.083 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:18.085 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:18.096 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":683,"stamp":"2024-09-18T14:28:16.265913+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23083,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14573,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620528,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123152,"statfs":{"total":193273528320,"available":192638107648,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1332,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":666,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":10581,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:12.483106+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:12.483106+0000","last_peered":"2024-09-18T14:28:12.483106+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:12.483106+0000","last_undegraded":"2024-09-18T14:28:12.483106+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4958,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2011,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2479},{"shards":"3","objects":2011}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":23083,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14573,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051625,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182442,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:27:16 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.53300000000000003,"15min":0.55000000000000004},"min":{"1min":0.217,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.76700000000000002,"5min":6.758,"15min":6.758},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.60899999999999999,"15min":0.60799999999999998},"min":{"1min":0.33400000000000002,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.78400000000000003,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.73799999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.628},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.96399999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705931,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.53200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:21.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:21.424 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:21.427 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:21.439 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2024-09-18T14:28:20.266827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":22323,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14953,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624688,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188118992,"statfs":{"total":193273528320,"available":192633847808,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003013"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":11722,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:17.483547+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:17.483547+0000","last_peered":"2024-09-18T14:28:17.483547+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:17.483547+0000","last_undegraded":"2024-09-18T14:28:17.483547+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4198,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2099},{"shards":"3","objects":2391}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":22323,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14953,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051626,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182443,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705933,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.72399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.623},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67800000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314424,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057416,"statfs":{"total":96636764160,"available":96314793984,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:21.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:21.754 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:21.757 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:21.767 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":685,"stamp":"2024-09-18T14:28:20.266827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":22323,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14953,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624688,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188118992,"statfs":{"total":193273528320,"available":192633847808,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003013"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":11722,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:17.483547+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:17.483547+0000","last_peered":"2024-09-18T14:28:17.483547+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:17.483547+0000","last_undegraded":"2024-09-18T14:28:17.483547+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4198,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2099},{"shards":"3","objects":2391}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":22323,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14953,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051626,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182443,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705933,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310264,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061576,"statfs":{"total":96636764160,"available":96319053824,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.72399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.623},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67800000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314424,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057416,"statfs":{"total":96636764160,"available":96314793984,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:21.768 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:21.768 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:21.768 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:21.768 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:21.769 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:21.769 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:21.769 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:28:21.769 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 11722, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:17.483547+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:28:17.483547+0000', 'last_peered': '2024-09-18T14:28:17.483547+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:28:17.483547+0000', 'last_undegraded': '2024-09-18T14:28:17.483547+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4198, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2391, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2099}, {'shards': '3', 'objects': 2391}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:21.769 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:21.769 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:21.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:22.161 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:22.161 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1313,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":639680512,"bytes_avail":192633847808,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":22323,"misplaced_total":72472,"misplaced_ratio":0.30802240865437686,"recovering_objects_per_sec":60,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:22.175 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:22.175 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:22.483 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:22.486 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:22.500 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":686,"stamp":"2024-09-18T14:28:22.267352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":22323,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14953,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612208,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131472,"statfs":{"total":193273528320,"available":192646627328,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1456,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":728,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003160"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":11722,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:17.483547+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:17.483547+0000","last_peered":"2024-09-18T14:28:17.483547+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:17.483547+0000","last_undegraded":"2024-09-18T14:28:17.483547+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4198,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2391,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2099},{"shards":"3","objects":2391}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":22323,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14953,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051626,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182444,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705934,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.621},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705932,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314424,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057416,"statfs":{"total":96636764160,"available":96314793984,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.72599999999999998,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.38800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.73699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:25.502 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:25.825 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:25.828 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:25.842 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2024-09-18T14:28:24.267842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":21471,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15379,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595568,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148112,"statfs":{"total":193273528320,"available":192663666688,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2308,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003081"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":12999,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:22.493470+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:22.493470+0000","last_peered":"2024-09-18T14:28:22.493470+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:22.493470+0000","last_undegraded":"2024-09-18T14:28:22.493470+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3346,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2817,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1673},{"shards":"3","objects":2817}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":21471,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15379,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051627,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182444,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705934,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.621},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.35799999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.441},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:25.843 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:26.150 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:26.165 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":687,"stamp":"2024-09-18T14:28:24.267842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":21471,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15379,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595568,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148112,"statfs":{"total":193273528320,"available":192663666688,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2308,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1154,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003081"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":12999,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:22.493470+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:22.493470+0000","last_peered":"2024-09-18T14:28:22.493470+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:22.493470+0000","last_undegraded":"2024-09-18T14:28:22.493470+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3346,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2817,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1673},{"shards":"3","objects":2817}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":21471,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15379,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051627,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182444,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705934,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.621},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.35799999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.441},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:26.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:26.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:26.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:26.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:26.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:26.167 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:26.167 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:28:26.167 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 12999, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:22.493470+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:28:22.493470+0000', 'last_peered': '2024-09-18T14:28:22.493470+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:28:22.493470+0000', 'last_undegraded': '2024-09-18T14:28:22.493470+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3346, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2817, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1673}, {'shards': '3', 'objects': 2817}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:26.167 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:26.167 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:26.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:26.546 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:26.546 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1317,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609861632,"bytes_avail":192663666688,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":21471,"misplaced_total":72472,"misplaced_ratio":0.29626614416602276,"recovering_objects_per_sec":96,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1154,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:26.566 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:26.566 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:26.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:26.880 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:26.892 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":688,"stamp":"2024-09-18T14:28:26.268592+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":21471,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15379,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595568,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148112,"statfs":{"total":193273528320,"available":192663666688,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1612,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":806,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003337"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":12999,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:22.493470+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:22.493470+0000","last_peered":"2024-09-18T14:28:22.493470+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:22.493470+0000","last_undegraded":"2024-09-18T14:28:22.493470+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3346,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2817,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1673},{"shards":"3","objects":2817}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":21471,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15379,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051627,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182444,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705934,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.65900000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.52000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.621},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705933,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.35799999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.441},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:29.894 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:30.216 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:30.219 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:30.232 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":689,"stamp":"2024-09-18T14:28:28.269083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":20511,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15859,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":599728,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":598844,"kb_avail":188143952,"statfs":{"total":193273528320,"available":192659406848,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":613216944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2572,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1286,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003170"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":14440,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:27.494265+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:27.494265+0000","last_peered":"2024-09-18T14:28:27.494265+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:27.494265+0000","last_undegraded":"2024-09-18T14:28:27.494265+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2386,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1193},{"shards":"3","objects":3297}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":20511,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15859,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051628,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705935,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53500000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.80000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.76400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:30.233 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:30.546 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:30.550 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:30.565 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2024-09-18T14:28:30.269680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":20511,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15859,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":599728,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":598844,"kb_avail":188143952,"statfs":{"total":193273528320,"available":192659406848,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":613216944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1812,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":14440,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:27.494265+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:27.494265+0000","last_peered":"2024-09-18T14:28:27.494265+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:27.494265+0000","last_undegraded":"2024-09-18T14:28:27.494265+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2386,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1193},{"shards":"3","objects":3297}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":20511,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15859,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051628,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705935,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53500000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.80000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.76400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:30.565 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:30.565 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:30.566 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:30.566 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:30.566 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:30.566 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:30.566 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:28:30.566 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 14440, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:27.494265+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:28:27.494265+0000', 'last_peered': '2024-09-18T14:28:27.494265+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:28:27.494265+0000', 'last_undegraded': '2024-09-18T14:28:27.494265+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2386, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3297, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1193}, {'shards': '3', 'objects': 3297}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:30.566 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:30.566 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:30.567 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:30.948 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:30.948 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1321,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":614121472,"bytes_avail":192659406848,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":20511,"misplaced_total":72472,"misplaced_ratio":0.28301964896787724,"recovering_objects_per_sec":75,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:30.959 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:30.959 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:31.271 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:31.285 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":690,"stamp":"2024-09-18T14:28:30.269680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":20511,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15859,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":599728,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":598844,"kb_avail":188143952,"statfs":{"total":193273528320,"available":192659406848,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":613216944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1812,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003294"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":14440,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:27.494265+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:27.494265+0000","last_peered":"2024-09-18T14:28:27.494265+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:27.494265+0000","last_undegraded":"2024-09-18T14:28:27.494265+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2386,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3297,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1193},{"shards":"3","objects":3297}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":20511,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15859,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051628,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182445,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705935,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297784,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074056,"statfs":{"total":96636764160,"available":96331833344,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.46899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53500000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705934,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.80000000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.76400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:34.286 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:34.610 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:34.613 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:34.626 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2024-09-18T14:28:34.270972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":19325,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16452,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603888,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139792,"statfs":{"total":193273528320,"available":192655147008,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2998,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":16218,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:32.494999+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:32.494999+0000","last_peered":"2024-09-18T14:28:32.494999+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:32.494999+0000","last_undegraded":"2024-09-18T14:28:32.494999+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1200,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3890,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":600},{"shards":"3","objects":3890}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":19325,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16452,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051629,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182446,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.629}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.54400000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.83799999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.502}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:34.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:34.942 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:34.946 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:34.960 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2024-09-18T14:28:34.270972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":19325,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16452,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603888,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139792,"statfs":{"total":193273528320,"available":192655147008,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2998,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":16218,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:32.494999+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:32.494999+0000","last_peered":"2024-09-18T14:28:32.494999+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:32.494999+0000","last_undegraded":"2024-09-18T14:28:32.494999+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1200,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3890,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":600},{"shards":"3","objects":3890}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":19325,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16452,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051629,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182446,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.629}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.54400000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.83799999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.502}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:34.961 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:34.961 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4636, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543555+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:38.543555+0000', 'last_peered': '2024-09-18T14:27:38.543555+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:38.543555+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:34.961 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:34.961 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4605, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.543598+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:38.543598+0000', 'last_peered': '2024-09-18T14:27:38.543598+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:38.543598+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:34.961 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:34.961 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4502, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.549726+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:38.549726+0000', 'last_peered': '2024-09-18T14:27:38.549726+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:38.549726+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:34.961 INFO:tasks.osd_recovery.ceph_manager:PG 1.1 is not active+clean 2024-09-18T14:28:34.961 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.1', 'version': "17'4490", 'reported_seq': 16218, 'reported_epoch': 37, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:32.494999+0000', 'last_change': '2024-09-18T14:27:38.508501+0000', 'last_active': '2024-09-18T14:28:32.494999+0000', 'last_peered': '2024-09-18T14:28:32.494999+0000', 'last_clean': '2024-09-18T14:23:58.513183+0000', 'last_became_active': '2024-09-18T14:24:59.560146+0000', 'last_became_peered': '2024-09-18T14:24:59.560146+0000', 'last_unstale': '2024-09-18T14:28:32.494999+0000', 'last_undegraded': '2024-09-18T14:28:32.494999+0000', 'last_fullsized': '2024-09-18T14:24:59.417290+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4490, 'log_dups_size': 0, 'ondisk_log_size': 4490, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000622711, 'stat_sum': {'num_bytes': 17605, 'num_objects': 4490, 'num_object_clones': 0, 'num_object_copies': 8980, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1200, 'num_objects_unfound': 0, 'num_objects_dirty': 4490, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4490, 'num_write_kb': 4502, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3890, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 600}, {'shards': '3', 'objects': 3890}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:34.962 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:34.962 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4571, 'reported_epoch': 36, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:38.546902+0000', 'last_change': '2024-09-18T14:24:48.467916+0000', 'last_active': '2024-09-18T14:27:38.546902+0000', 'last_peered': '2024-09-18T14:27:38.546902+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:27:38.546902+0000', 'last_undegraded': '2024-09-18T14:24:48.455174+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4531, 'num_objects_misplaced': 4531, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4531}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:34.962 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:35.368 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:35.369 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 18125/72472 objects degraded (25.010%), 4 pgs degraded, 5 pgs undersized","count":9},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1326,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":37,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":5},"pgmap":{"pgs_by_state":[{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":4},{"state_name":"active+clean","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618381312,"bytes_avail":192655147008,"bytes_total":193273528320,"degraded_objects":18125,"degraded_total":72472,"degraded_ratio":0.25009658902748649,"misplaced_objects":19325,"misplaced_total":72472,"misplaced_ratio":0.26665470802516833,"recovering_objects_per_sec":124,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:35.386 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:35.387 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:35.702 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:35.705 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:35.717 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":692,"stamp":"2024-09-18T14:28:34.270972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":19325,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16452,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603888,"kb_used_data":864,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139792,"statfs":{"total":193273528320,"available":192655147008,"internally_reserved":0,"allocated":884736,"data_stored":435817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2998,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1499,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003619"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4636,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543555+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:38.543555+0000","last_peered":"2024-09-18T14:27:38.543555+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:38.543555+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4605,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.543598+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:38.543598+0000","last_peered":"2024-09-18T14:27:38.543598+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:38.543598+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4502,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.549726+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:38.549726+0000","last_peered":"2024-09-18T14:27:38.549726+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:38.549726+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4563,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545854+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:27:38.545854+0000","last_peered":"2024-09-18T14:27:38.545854+0000","last_clean":"2024-09-18T14:27:38.545854+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:27:38.545854+0000","last_undegraded":"2024-09-18T14:27:38.545854+0000","last_fullsized":"2024-09-18T14:27:38.545854+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4526,"reported_epoch":37,"state":"active+clean","last_fresh":"2024-09-18T14:27:39.555275+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:27:39.555275+0000","last_peered":"2024-09-18T14:27:39.555275+0000","last_clean":"2024-09-18T14:27:39.555275+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:27:39.555275+0000","last_undegraded":"2024-09-18T14:27:39.555275+0000","last_fullsized":"2024-09-18T14:27:39.555275+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4676,"reported_epoch":36,"state":"active+clean","last_fresh":"2024-09-18T14:27:38.545754+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:27:38.545754+0000","last_peered":"2024-09-18T14:27:38.545754+0000","last_clean":"2024-09-18T14:27:38.545754+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:27:38.545754+0000","last_undegraded":"2024-09-18T14:27:38.545754+0000","last_fullsized":"2024-09-18T14:27:38.545754+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":16218,"reported_epoch":37,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:32.494999+0000","last_change":"2024-09-18T14:27:38.508501+0000","last_active":"2024-09-18T14:28:32.494999+0000","last_peered":"2024-09-18T14:28:32.494999+0000","last_clean":"2024-09-18T14:23:58.513183+0000","last_became_active":"2024-09-18T14:24:59.560146+0000","last_became_peered":"2024-09-18T14:24:59.560146+0000","last_unstale":"2024-09-18T14:28:32.494999+0000","last_undegraded":"2024-09-18T14:28:32.494999+0000","last_fullsized":"2024-09-18T14:24:59.417290+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:43:14.560023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1200,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3890,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":600},{"shards":"3","objects":3890}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4571,"reported_epoch":36,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:38.546902+0000","last_change":"2024-09-18T14:24:48.467916+0000","last_active":"2024-09-18T14:27:38.546902+0000","last_peered":"2024-09-18T14:27:38.546902+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:27:38.546902+0000","last_undegraded":"2024-09-18T14:24:48.455174+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4531,"num_objects_misplaced":4531,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4531}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18125,"num_objects_misplaced":19325,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16452,"num_bytes_recovered":104966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":909312,"data_stored":708388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":11,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051629,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182446,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":218038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.629}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.54400000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.83799999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705936,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301944,"kb_used_data":432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069896,"statfs":{"total":96636764160,"available":96327573504,"internally_reserved":0,"allocated":442368,"data_stored":217779,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.502}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57599999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":135168,"data_stored":104947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:38.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:39.050 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:39.053 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:39.066 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2024-09-18T14:28:38.272091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22598,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17081,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612276,"kb_used_data":920,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131404,"statfs":{"total":193273528320,"available":192646557696,"internally_reserved":0,"allocated":942080,"data_stored":471937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4531,"num_objects_misplaced":2087,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002411"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.940013+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:39.940013+0000","last_peered":"2024-09-18T14:27:39.940013+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:39.940013+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.547376+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:39.547376+0000","last_peered":"2024-09-18T14:27:39.547376+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:39.547376+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.941149+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:39.941149+0000","last_peered":"2024-09-18T14:27:39.941149+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:39.941149+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4660,"reported_epoch":38,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:36.380493+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:36.380493+0000","last_peered":"2024-09-18T14:28:36.380493+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:36.380493+0000","last_undegraded":"2024-09-18T14:28:36.380493+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9004,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":29},{"shards":"2,3","objects":4502}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22598,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17081,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051630,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182447,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705937,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306132,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065708,"statfs":{"total":96636764160,"available":96323284992,"internally_reserved":0,"allocated":466944,"data_stored":234457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.53400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.46999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.48799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705937,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306144,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065696,"statfs":{"total":96636764160,"available":96323272704,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.69699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:39.067 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:39.365 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:39.368 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:39.379 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2024-09-18T14:28:38.272091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22598,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17081,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612276,"kb_used_data":920,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131404,"statfs":{"total":193273528320,"available":192646557696,"internally_reserved":0,"allocated":942080,"data_stored":471937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4531,"num_objects_misplaced":2087,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002411"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.940013+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:39.940013+0000","last_peered":"2024-09-18T14:27:39.940013+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:39.940013+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.547376+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:39.547376+0000","last_peered":"2024-09-18T14:27:39.547376+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:39.547376+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.941149+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:39.941149+0000","last_peered":"2024-09-18T14:27:39.941149+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:39.941149+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4660,"reported_epoch":38,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:36.380493+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:36.380493+0000","last_peered":"2024-09-18T14:28:36.380493+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:36.380493+0000","last_undegraded":"2024-09-18T14:28:36.380493+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9004,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":29},{"shards":"2,3","objects":4502}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22598,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17081,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051630,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182447,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705937,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306132,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065708,"statfs":{"total":96636764160,"available":96323284992,"internally_reserved":0,"allocated":466944,"data_stored":234457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.53400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.46999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.48799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705937,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306144,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065696,"statfs":{"total":96636764160,"available":96323272704,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.69699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:39.379 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:39.380 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4637, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:39.940013+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:27:39.940013+0000', 'last_peered': '2024-09-18T14:27:39.940013+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:27:39.940013+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:39.380 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:39.380 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4606, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:39.547376+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:39.547376+0000', 'last_peered': '2024-09-18T14:27:39.547376+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:39.547376+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:39.380 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:39.380 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4503, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:39.941149+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:27:39.941149+0000', 'last_peered': '2024-09-18T14:27:39.941149+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:27:39.941149+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:39.380 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:39.380 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 4660, 'reported_epoch': 38, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:36.380493+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:28:36.380493+0000', 'last_peered': '2024-09-18T14:28:36.380493+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:28:36.380493+0000', 'last_undegraded': '2024-09-18T14:28:36.380493+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 9004, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 29, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 29}, {'shards': '2,3', 'objects': 4502}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:39.381 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:39.755 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:39.755 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1330,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626970624,"bytes_avail":192646557696,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":22598,"misplaced_total":72472,"misplaced_ratio":0.31181697759134563,"recovering_objects_per_sec":152,"recovering_bytes_per_sec":1639,"recovering_keys_per_sec":0,"num_objects_recovered":1222,"num_bytes_recovered":13116,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:39.769 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:39.769 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:40.075 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:40.079 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:40.094 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":696,"stamp":"2024-09-18T14:28:38.272091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22598,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17081,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612276,"kb_used_data":920,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131404,"statfs":{"total":193273528320,"available":192646557696,"internally_reserved":0,"allocated":942080,"data_stored":471937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4531,"num_objects_misplaced":2087,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1222,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002411"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4637,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.940013+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:27:39.940013+0000","last_peered":"2024-09-18T14:27:39.940013+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:27:39.940013+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.547376+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:39.547376+0000","last_peered":"2024-09-18T14:27:39.547376+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:39.547376+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4503,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.941149+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:27:39.941149+0000","last_peered":"2024-09-18T14:27:39.941149+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:27:39.941149+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4660,"reported_epoch":38,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:36.380493+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:36.380493+0000","last_peered":"2024-09-18T14:28:36.380493+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:36.380493+0000","last_undegraded":"2024-09-18T14:28:36.380493+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9004,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":29},{"shards":"2,3","objects":4502}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22598,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17081,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051630,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182447,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705937,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306132,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065708,"statfs":{"total":96636764160,"available":96323284992,"internally_reserved":0,"allocated":466944,"data_stored":234457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.53400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.46999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.48799999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705937,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306144,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065696,"statfs":{"total":96636764160,"available":96323272704,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.69699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:43.097 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:43.420 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:43.423 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:43.437 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2024-09-18T14:28:42.273094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612280,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131400,"statfs":{"total":193273528320,"available":192646553600,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4531,"num_objects_misplaced":2751,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":890,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002122"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.547376+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:39.547376+0000","last_peered":"2024-09-18T14:27:39.547376+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:39.547376+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":5444,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:41.376933+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:41.376933+0000","last_peered":"2024-09-18T14:28:41.376933+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:41.376933+0000","last_undegraded":"2024-09-18T14:28:41.376933+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8482,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":290},{"shards":"2,3","objects":4241}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051630,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.81899999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705937,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306144,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065696,"statfs":{"total":96636764160,"available":96323272704,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.69699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:43.438 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:43.730 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:43.733 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:43.743 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":698,"stamp":"2024-09-18T14:28:42.273094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612280,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131400,"statfs":{"total":193273528320,"available":192646553600,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4531,"num_objects_misplaced":2751,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":890,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002122"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":37,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:27:39.547376+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:27:39.547376+0000","last_peered":"2024-09-18T14:27:39.547376+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:27:39.547376+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":5444,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:41.376933+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:41.376933+0000","last_peered":"2024-09-18T14:28:41.376933+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:41.376933+0000","last_undegraded":"2024-09-18T14:28:41.376933+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8482,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":290},{"shards":"2,3","objects":4241}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051630,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.81899999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705937,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306144,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065696,"statfs":{"total":96636764160,"available":96323272704,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.89100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.69699999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:43.744 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:43.744 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:43.745 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:43.745 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4606, 'reported_epoch': 37, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:27:39.547376+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:27:39.547376+0000', 'last_peered': '2024-09-18T14:27:39.547376+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:27:39.547376+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:43.745 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:43.745 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:43.745 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:43.745 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 5444, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:41.376933+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:28:41.376933+0000', 'last_peered': '2024-09-18T14:28:41.376933+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:28:41.376933+0000', 'last_undegraded': '2024-09-18T14:28:41.376933+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8482, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 290, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 290}, {'shards': '2,3', 'objects': 4241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:43.746 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:44.113 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:44.113 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1334,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626974720,"bytes_avail":192646553600,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":22076,"misplaced_total":72472,"misplaced_ratio":0.304614195827354,"recovering_objects_per_sec":111,"recovering_bytes_per_sec":1639,"recovering_keys_per_sec":0,"num_objects_recovered":890,"num_bytes_recovered":13116,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:44.130 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:44.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:44.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:44.435 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:44.447 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":699,"stamp":"2024-09-18T14:28:44.273690+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612272,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131408,"statfs":{"total":193273528320,"available":192646561792,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4531,"num_objects_misplaced":2751,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":890,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.199312"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":5444,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:41.376933+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:41.376933+0000","last_peered":"2024-09-18T14:28:41.376933+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:41.376933+0000","last_undegraded":"2024-09-18T14:28:41.376933+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8482,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":290},{"shards":"2,3","objects":4241}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051631,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.81899999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:47.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:47.762 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:47.765 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:47.775 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2024-09-18T14:28:46.274184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612272,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131408,"statfs":{"total":193273528320,"available":192646561792,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4531,"num_objects_misplaced":2751,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":890,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002660"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":5444,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:41.376933+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:41.376933+0000","last_peered":"2024-09-18T14:28:41.376933+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:41.376933+0000","last_undegraded":"2024-09-18T14:28:41.376933+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8482,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":290},{"shards":"2,3","objects":4241}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051631,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.81899999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:47.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:48.081 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:48.084 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:48.094 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":700,"stamp":"2024-09-18T14:28:46.274184+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612272,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131408,"statfs":{"total":193273528320,"available":192646561792,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4531,"num_objects_misplaced":2751,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":890,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002660"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":5444,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:41.376933+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:41.376933+0000","last_peered":"2024-09-18T14:28:41.376933+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:41.376933+0000","last_undegraded":"2024-09-18T14:28:41.376933+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8482,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":290,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":290},{"shards":"2,3","objects":4241}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":22076,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17342,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051631,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.74299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.81899999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67700000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705938,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:48.095 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:48.095 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:48.095 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:48.095 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:48.095 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:48.095 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:48.096 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:48.096 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 5444, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:41.376933+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:28:41.376933+0000', 'last_peered': '2024-09-18T14:28:41.376933+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:28:41.376933+0000', 'last_undegraded': '2024-09-18T14:28:41.376933+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8482, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 290, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 290}, {'shards': '2,3', 'objects': 4241}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:48.096 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:48.479 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:48.480 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1339,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626966528,"bytes_avail":192646561792,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":21526,"misplaced_total":72472,"misplaced_ratio":0.29702505795341649,"recovering_objects_per_sec":101,"recovering_bytes_per_sec":1172,"recovering_keys_per_sec":0,"num_objects_recovered":1136,"num_bytes_recovered":13116,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:48.494 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:48.494 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:48.791 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:48.804 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":701,"stamp":"2024-09-18T14:28:48.274813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":21526,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17617,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612272,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131408,"statfs":{"total":193273528320,"available":192646561792,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2272,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1136,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.188609"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":6269,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:46.388231+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:46.388231+0000","last_peered":"2024-09-18T14:28:46.388231+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:46.388231+0000","last_undegraded":"2024-09-18T14:28:46.388231+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7932,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":565},{"shards":"2,3","objects":3966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":21526,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17617,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051632,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182449,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705939,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.59499999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:51.806 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:52.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:52.104 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:52.119 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":702,"stamp":"2024-09-18T14:28:50.275571+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":21526,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17617,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612272,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131408,"statfs":{"total":193273528320,"available":192646561792,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1072,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":536,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003480"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":6269,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:46.388231+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:46.388231+0000","last_peered":"2024-09-18T14:28:46.388231+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:46.388231+0000","last_undegraded":"2024-09-18T14:28:46.388231+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7932,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":565,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":565},{"shards":"2,3","objects":3966}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":21526,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17617,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051632,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182449,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705939,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:27:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.49099999999999999,"15min":0.47699999999999998},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":2.097,"15min":2.7770000000000001},"min":{"1min":0.34899999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.83699999999999997,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.59499999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:52.120 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:52.433 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:52.437 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:52.448 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2024-09-18T14:28:52.276155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616432,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188127248,"statfs":{"total":193273528320,"available":192642301952,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1630,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003573"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":7106,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:51.393530+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:51.393530+0000","last_peered":"2024-09-18T14:28:51.393530+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:51.393530+0000","last_undegraded":"2024-09-18T14:28:51.393530+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7374,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":844},{"shards":"2,3","objects":3687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051632,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.502},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.621}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:52.449 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:52.449 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:52.449 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:52.449 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:52.449 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:52.449 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:52.449 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:52.449 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 7106, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:51.393530+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:28:51.393530+0000', 'last_peered': '2024-09-18T14:28:51.393530+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:28:51.393530+0000', 'last_undegraded': '2024-09-18T14:28:51.393530+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7374, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 844, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 844}, {'shards': '2,3', 'objects': 3687}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:52.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:52.820 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1343,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":631226368,"bytes_avail":192642301952,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":20968,"misplaced_total":72472,"misplaced_ratio":0.28932553261949445,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":815,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:52.834 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:52.834 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:53.124 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:53.127 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:53.138 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":703,"stamp":"2024-09-18T14:28:52.276155+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616432,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188127248,"statfs":{"total":193273528320,"available":192642301952,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1630,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":815,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003573"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":7106,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:51.393530+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:51.393530+0000","last_peered":"2024-09-18T14:28:51.393530+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:51.393530+0000","last_undegraded":"2024-09-18T14:28:51.393530+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7374,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":844},{"shards":"2,3","objects":3687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051632,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.502},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.621}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705939,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.60299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.46000000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.436}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:56.140 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:56.450 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:56.453 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:56.463 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":705,"stamp":"2024-09-18T14:28:56.277812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004121"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":7106,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:51.393530+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:51.393530+0000","last_peered":"2024-09-18T14:28:51.393530+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:51.393530+0000","last_undegraded":"2024-09-18T14:28:51.393530+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7374,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":844},{"shards":"2,3","objects":3687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051633,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.502},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.621}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.625},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58799999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.80200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:56.464 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:56.760 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:56.764 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:56.780 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":705,"stamp":"2024-09-18T14:28:56.277812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004121"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":7106,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:51.393530+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:51.393530+0000","last_peered":"2024-09-18T14:28:51.393530+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:51.393530+0000","last_undegraded":"2024-09-18T14:28:51.393530+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7374,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":844},{"shards":"2,3","objects":3687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051633,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.502},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.621}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.625},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58799999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.80200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:28:56.780 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:28:56.781 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:56.781 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:28:56.781 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:28:56.781 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:28:56.781 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:56.781 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:28:56.781 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 7106, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:51.393530+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:28:51.393530+0000', 'last_peered': '2024-09-18T14:28:51.393530+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:28:51.393530+0000', 'last_undegraded': '2024-09-18T14:28:51.393530+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7374, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 844, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 844}, {'shards': '2,3', 'objects': 3687}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:28:56.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:28:57.134 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:57.134 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1348,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635486208,"bytes_avail":192638042112,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":20968,"misplaced_total":72472,"misplaced_ratio":0.28932553261949445,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:28:57.152 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:28:57.152 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:28:57.474 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:28:57.477 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:28:57.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":705,"stamp":"2024-09-18T14:28:56.277812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1108,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004121"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":7106,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:51.393530+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:51.393530+0000","last_peered":"2024-09-18T14:28:51.393530+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:51.393530+0000","last_undegraded":"2024-09-18T14:28:51.393530+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7374,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":844,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":844},{"shards":"2,3","objects":3687}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20968,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17896,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051633,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182450,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.502},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.621}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45200000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705940,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.73299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:27:52 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.52100000000000002,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.74299999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.625},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.28399999999999997,"5min":0.247,"15min":0.247},"max":{"1min":0.96199999999999997,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58799999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.80200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:00.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:00.794 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:00.798 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:00.808 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2024-09-18T14:29:00.278620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20374,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18193,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1152,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003806"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":7998,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:56.393682+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:56.393682+0000","last_peered":"2024-09-18T14:28:56.393682+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:56.393682+0000","last_undegraded":"2024-09-18T14:28:56.393682+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6780,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1141},{"shards":"2,3","objects":3390}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20374,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18193,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051634,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705941,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.72699999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79400000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.314},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.37}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:00.809 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:01.151 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:01.154 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:01.165 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2024-09-18T14:29:00.278620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20374,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18193,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1152,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003806"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":7998,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:56.393682+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:56.393682+0000","last_peered":"2024-09-18T14:28:56.393682+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:56.393682+0000","last_undegraded":"2024-09-18T14:28:56.393682+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6780,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1141},{"shards":"2,3","objects":3390}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20374,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18193,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051634,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705941,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.72699999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79400000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.314},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.37}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:01.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:01.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:01.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:01.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:01.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:01.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:01.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:01.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 7998, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:28:56.393682+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:28:56.393682+0000', 'last_peered': '2024-09-18T14:28:56.393682+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:28:56.393682+0000', 'last_undegraded': '2024-09-18T14:28:56.393682+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6780, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1141, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1141}, {'shards': '2,3', 'objects': 3390}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:01.167 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:01.542 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:01.542 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1352,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635486208,"bytes_avail":192638042112,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":20374,"misplaced_total":72472,"misplaced_ratio":0.2811292637156419,"recovering_objects_per_sec":47,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":576,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:01.563 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:01.564 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:01.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:01.861 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:01.877 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":707,"stamp":"2024-09-18T14:29:00.278620+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20374,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18193,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1152,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003806"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":7998,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:28:56.393682+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:28:56.393682+0000","last_peered":"2024-09-18T14:28:56.393682+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:28:56.393682+0000","last_undegraded":"2024-09-18T14:28:56.393682+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6780,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1141},{"shards":"2,3","objects":3390}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":20374,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18193,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051634,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182451,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705941,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.72699999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79400000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705941,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.314},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.37}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.53100000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:04.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:05.197 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:05.200 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:05.211 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-09-18T14:29:04.279854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19766,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18497,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1202,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003699"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":8909,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:01.394479+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:01.394479+0000","last_peered":"2024-09-18T14:29:01.394479+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:01.394479+0000","last_undegraded":"2024-09-18T14:29:01.394479+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6172,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1445,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1445},{"shards":"2,3","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19766,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18497,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051635,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705942,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.67300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.622}]}]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.38},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:05.212 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:05.528 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:05.532 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:05.544 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-09-18T14:29:04.279854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19766,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18497,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1202,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003699"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":8909,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:01.394479+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:01.394479+0000","last_peered":"2024-09-18T14:29:01.394479+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:01.394479+0000","last_undegraded":"2024-09-18T14:29:01.394479+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6172,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1445,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1445},{"shards":"2,3","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19766,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18497,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051635,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705942,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.67300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.622}]}]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.38},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:05.544 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:05.545 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:05.545 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:05.545 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:05.545 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:05.545 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:05.545 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:05.545 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 8909, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:01.394479+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:29:01.394479+0000', 'last_peered': '2024-09-18T14:29:01.394479+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:29:01.394479+0000', 'last_undegraded': '2024-09-18T14:29:01.394479+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6172, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1445, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1445}, {'shards': '2,3', 'objects': 3086}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:05.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:05.908 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:05.908 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1356,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635486208,"bytes_avail":192638042112,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":19766,"misplaced_total":72472,"misplaced_ratio":0.27273981675681641,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":601,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:05.924 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:05.924 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:06.217 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:06.219 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:06.235 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":709,"stamp":"2024-09-18T14:29:04.279854+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19766,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18497,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1202,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":601,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003699"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":8909,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:01.394479+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:01.394479+0000","last_peered":"2024-09-18T14:29:01.394479+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:01.394479+0000","last_undegraded":"2024-09-18T14:29:01.394479+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6172,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1445,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1445},{"shards":"2,3","objects":3086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19766,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18497,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051635,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182452,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705942,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.85499999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.67300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.622}]}]},{"osd":0,"up_from":9,"seq":38654705942,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.38},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.35399999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.72699999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68400000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:09.240 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:09.538 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:09.542 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:09.553 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2024-09-18T14:29:08.281031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19120,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18820,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1848,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":924,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003218"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":9878,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:06.403084+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:06.403084+0000","last_peered":"2024-09-18T14:29:06.403084+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:06.403084+0000","last_undegraded":"2024-09-18T14:29:06.403084+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5526,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1768},{"shards":"2,3","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19120,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18820,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051636,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182453,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705943,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.80600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73499999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.66400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:09.554 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:09.872 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:09.875 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:09.887 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":711,"stamp":"2024-09-18T14:29:08.281031+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19120,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18820,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1848,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":924,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003218"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":9878,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:06.403084+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:06.403084+0000","last_peered":"2024-09-18T14:29:06.403084+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:06.403084+0000","last_undegraded":"2024-09-18T14:29:06.403084+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5526,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1768},{"shards":"2,3","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19120,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18820,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051636,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182453,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705943,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.80600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73499999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.66400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:09.888 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:09.888 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:09.888 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:09.888 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:09.888 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:09.888 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:09.888 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:09.888 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 9878, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:06.403084+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:29:06.403084+0000', 'last_peered': '2024-09-18T14:29:06.403084+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:29:06.403084+0000', 'last_undegraded': '2024-09-18T14:29:06.403084+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5526, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1768, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1768}, {'shards': '2,3', 'objects': 2763}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:09.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:10.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:10.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1361,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635486208,"bytes_avail":192638042112,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":19120,"misplaced_total":72472,"misplaced_ratio":0.26382602936306437,"recovering_objects_per_sec":76,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":924,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:10.283 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:10.283 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:10.598 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:10.601 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:10.616 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":712,"stamp":"2024-09-18T14:29:10.281628+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19120,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18820,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620592,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188123088,"statfs":{"total":193273528320,"available":192638042112,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1254,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":627,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003435"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":9878,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:06.403084+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:06.403084+0000","last_peered":"2024-09-18T14:29:06.403084+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:06.403084+0000","last_undegraded":"2024-09-18T14:29:06.403084+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5526,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1768,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1768},{"shards":"2,3","objects":2763}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":19120,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18820,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051636,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182453,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705943,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.89200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.80600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:28:04 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.55900000000000005,"15min":0.54000000000000004},"min":{"1min":0.35899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73499999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.66400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:13.618 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:13.927 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:13.930 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:13.940 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2024-09-18T14:29:12.281972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":18410,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19175,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":608112,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135568,"statfs":{"total":193273528320,"available":192650821632,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1964,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":982,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003352"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":10943,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:11.398504+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:11.398504+0000","last_peered":"2024-09-18T14:29:11.398504+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:11.398504+0000","last_undegraded":"2024-09-18T14:29:11.398504+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4816,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2123},{"shards":"2,3","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":18410,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19175,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051636,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705944,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.377}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.66400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:13.941 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:14.268 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:14.288 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":713,"stamp":"2024-09-18T14:29:12.281972+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":18410,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19175,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":608112,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135568,"statfs":{"total":193273528320,"available":192650821632,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1964,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":982,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003352"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":10943,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:11.398504+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:11.398504+0000","last_peered":"2024-09-18T14:29:11.398504+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:11.398504+0000","last_undegraded":"2024-09-18T14:29:11.398504+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4816,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2123},{"shards":"2,3","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":18410,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19175,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051636,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705944,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.377}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705943,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310296,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061544,"statfs":{"total":96636764160,"available":96319021056,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.66400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:28:03 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.41099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98799999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.36099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:14.289 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:14.289 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:14.289 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:14.289 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:14.289 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:14.289 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:14.289 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:14.290 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 10943, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:11.398504+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:29:11.398504+0000', 'last_peered': '2024-09-18T14:29:11.398504+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:29:11.398504+0000', 'last_undegraded': '2024-09-18T14:29:11.398504+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4816, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2123, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2123}, {'shards': '2,3', 'objects': 2408}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:14.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:14.665 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:14.665 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1365,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609927168,"bytes_avail":192663601152,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":18410,"misplaced_total":72472,"misplaced_ratio":0.25402914228943591,"recovering_objects_per_sec":56,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":678,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:14.679 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:14.679 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:14.979 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:14.982 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:14.992 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":714,"stamp":"2024-09-18T14:29:14.282479+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":18410,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19175,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595632,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148048,"statfs":{"total":193273528320,"available":192663601152,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1356,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":678,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003242"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":10943,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:11.398504+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:11.398504+0000","last_peered":"2024-09-18T14:29:11.398504+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:11.398504+0000","last_undegraded":"2024-09-18T14:29:11.398504+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4816,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2123},{"shards":"2,3","objects":2408}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":18410,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19175,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051637,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705944,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.377}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.60399999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705944,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.34000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.35899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:17.994 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:18.316 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:18.319 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:18.329 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":716,"stamp":"2024-09-18T14:29:18.283524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":17644,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19558,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595632,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148048,"statfs":{"total":193273528320,"available":192663601152,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":12092,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:16.410059+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:16.410059+0000","last_peered":"2024-09-18T14:29:16.410059+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:16.410059+0000","last_undegraded":"2024-09-18T14:29:16.410059+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4050,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2506,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2506},{"shards":"2,3","objects":2025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":17644,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19558,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051638,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.63200000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.377},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:18.330 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:18.645 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:18.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":716,"stamp":"2024-09-18T14:29:18.283524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":17644,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19558,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595632,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148048,"statfs":{"total":193273528320,"available":192663601152,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":12092,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:16.410059+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:16.410059+0000","last_peered":"2024-09-18T14:29:16.410059+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:16.410059+0000","last_undegraded":"2024-09-18T14:29:16.410059+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4050,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2506,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2506},{"shards":"2,3","objects":2025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":17644,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19558,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051638,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.63200000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.377},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:18.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:18.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:18.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:18.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:18.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:18.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:18.660 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:18.660 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 12092, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:16.410059+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:29:16.410059+0000', 'last_peered': '2024-09-18T14:29:16.410059+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:29:16.410059+0000', 'last_undegraded': '2024-09-18T14:29:16.410059+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4050, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2506, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2506}, {'shards': '2,3', 'objects': 2025}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:18.661 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:19.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:19.043 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1369,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":609927168,"bytes_avail":192663601152,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":17644,"misplaced_total":72472,"misplaced_ratio":0.24345954299591566,"recovering_objects_per_sec":88,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1061,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":12,"modified":"2024-09-18T14:25:58.219937+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:19.056 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:19.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:19.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:19.376 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:19.389 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":716,"stamp":"2024-09-18T14:29:18.283524+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":17644,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19558,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595632,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188148048,"statfs":{"total":193273528320,"available":192663601152,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1061,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003093"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":12092,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:16.410059+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:16.410059+0000","last_peered":"2024-09-18T14:29:16.410059+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:16.410059+0000","last_undegraded":"2024-09-18T14:29:16.410059+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4050,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2506,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2506},{"shards":"2,3","objects":2025}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":17644,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19558,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051638,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.52400000000000002,"15min":0.54500000000000004},"min":{"1min":0.26900000000000002,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.97299999999999998,"5min":6.758,"15min":6.758},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.60699999999999998,"15min":0.60599999999999998},"min":{"1min":0.36499999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.85999999999999999,"5min":6.7859999999999996,"15min":6.7859999999999996},"last":0.47299999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.40400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.63200000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.377},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:22.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:22.697 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:22.701 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:22.712 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":718,"stamp":"2024-09-18T14:29:22.284508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":599792,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":598844,"kb_avail":188143888,"statfs":{"total":193273528320,"available":192659341312,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":613216944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2326,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":13367,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:21.408173+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:21.408173+0000","last_peered":"2024-09-18T14:29:21.408173+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:21.408173+0000","last_undegraded":"2024-09-18T14:29:21.408173+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3200,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2931},{"shards":"2,3","objects":1600}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051638,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.70199999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.377},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:22.713 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:23.033 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:23.036 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:23.047 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":718,"stamp":"2024-09-18T14:29:22.284508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":599792,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":598844,"kb_avail":188143888,"statfs":{"total":193273528320,"available":192659341312,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":613216944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2326,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":13367,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:21.408173+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:21.408173+0000","last_peered":"2024-09-18T14:29:21.408173+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:21.408173+0000","last_undegraded":"2024-09-18T14:29:21.408173+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3200,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2931},{"shards":"2,3","objects":1600}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051638,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.70199999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.377},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:23.048 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:23.049 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:23.049 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:23.049 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:23.049 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:23.049 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:23.049 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:23.049 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 13367, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:21.408173+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:29:21.408173+0000', 'last_peered': '2024-09-18T14:29:21.408173+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:29:21.408173+0000', 'last_undegraded': '2024-09-18T14:29:21.408173+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3200, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2931, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2931}, {'shards': '2,3', 'objects': 1600}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:23.049 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:23.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:23.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1374,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":614187008,"bytes_avail":192659341312,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":16794,"misplaced_total":72472,"misplaced_ratio":0.23173087537255768,"recovering_objects_per_sec":96,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1163,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-18T14:29:20.284127+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T14:29:23.449 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:23.449 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:23.775 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:23.779 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:23.792 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":718,"stamp":"2024-09-18T14:29:22.284508+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":599792,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":598844,"kb_avail":188143888,"statfs":{"total":193273528320,"available":192659341312,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":613216944},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2326,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1163,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002880"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":13367,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:21.408173+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:21.408173+0000","last_peered":"2024-09-18T14:29:21.408173+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:21.408173+0000","last_undegraded":"2024-09-18T14:29:21.408173+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3200,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2931},{"shards":"2,3","objects":1600}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051638,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.70199999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705945,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297816,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94074024,"statfs":{"total":96636764160,"available":96331800576,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:28:20 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45700000000000002,"15min":0.52900000000000003},"min":{"1min":0.26000000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66600000000000004,"5min":0.88600000000000001,"15min":0.95299999999999996},"last":0.377},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.55500000000000005,"15min":0.65000000000000002},"min":{"1min":0.31900000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.50800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.62}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:26.794 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:27.137 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:27.140 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:27.153 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":720,"stamp":"2024-09-18T14:29:26.285550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603952,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139728,"statfs":{"total":193273528320,"available":192655081472,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":13367,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:21.408173+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:21.408173+0000","last_peered":"2024-09-18T14:29:21.408173+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:21.408173+0000","last_undegraded":"2024-09-18T14:29:21.408173+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3200,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2931},{"shards":"2,3","objects":1600}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051639,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.70199999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.3959999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.542}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.371}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.499},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:27.154 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:27.458 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:27.464 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:27.472 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":720,"stamp":"2024-09-18T14:29:26.285550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603952,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139728,"statfs":{"total":193273528320,"available":192655081472,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":13367,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:21.408173+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:21.408173+0000","last_peered":"2024-09-18T14:29:21.408173+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:21.408173+0000","last_undegraded":"2024-09-18T14:29:21.408173+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3200,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2931},{"shards":"2,3","objects":1600}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051639,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.70199999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.3959999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.542}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.371}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.499},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:27.473 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:27.473 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:27.473 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:27.474 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:27.474 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:27.474 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:27.474 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:27.474 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 13367, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:21.408173+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:29:21.408173+0000', 'last_peered': '2024-09-18T14:29:21.408173+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:29:21.408173+0000', 'last_undegraded': '2024-09-18T14:29:21.408173+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3200, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2931, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2931}, {'shards': '2,3', 'objects': 1600}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:27.474 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:27.866 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:27.866 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1378,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618446848,"bytes_avail":192655081472,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":16794,"misplaced_total":72472,"misplaced_ratio":0.23173087537255768,"recovering_objects_per_sec":67,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":808,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-18T14:29:20.284127+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T14:29:27.879 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:27.879 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:28.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:28.209 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":720,"stamp":"2024-09-18T14:29:26.285550+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603952,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139728,"statfs":{"total":193273528320,"available":192655081472,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1616,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":808,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003071"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":13367,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:21.408173+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:21.408173+0000","last_peered":"2024-09-18T14:29:21.408173+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:21.408173+0000","last_undegraded":"2024-09-18T14:29:21.408173+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3200,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2931},{"shards":"2,3","objects":1600}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":16794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19983,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051639,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182456,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63400000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51600000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.70199999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705946,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":1.3959999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":1.542}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.371}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.499},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:31.211 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:31.539 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:31.542 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:31.558 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2024-09-18T14:29:30.286778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":15832,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20464,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603952,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139728,"statfs":{"total":193273528320,"available":192655081472,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1812,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":14810,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:26.418603+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:26.418603+0000","last_peered":"2024-09-18T14:29:26.418603+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:26.418603+0000","last_undegraded":"2024-09-18T14:29:26.418603+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2238,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3412},{"shards":"2,3","objects":1119}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":15832,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20464,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051640,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705947,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.74199999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.58199999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:31.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:31.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:31.891 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":722,"stamp":"2024-09-18T14:29:30.286778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":15832,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20464,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603952,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139728,"statfs":{"total":193273528320,"available":192655081472,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1812,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003254"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":14810,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:26.418603+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:26.418603+0000","last_peered":"2024-09-18T14:29:26.418603+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:26.418603+0000","last_undegraded":"2024-09-18T14:29:26.418603+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2238,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3412,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3412},{"shards":"2,3","objects":1119}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":15832,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20464,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051640,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182457,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705947,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63100000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.74199999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.58199999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:31.892 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:31.892 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:31.893 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:31.893 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:31.893 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:31.893 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:31.893 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:31.893 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 14810, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:26.418603+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:29:26.418603+0000', 'last_peered': '2024-09-18T14:29:26.418603+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:29:26.418603+0000', 'last_undegraded': '2024-09-18T14:29:26.418603+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2238, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3412, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3412}, {'shards': '2,3', 'objects': 1119}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:31.893 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:32.282 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:32.282 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1383,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":39,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":4},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618446848,"bytes_avail":192655081472,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":15832,"misplaced_total":72472,"misplaced_ratio":0.21845678330941606,"recovering_objects_per_sec":75,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":906,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":13,"modified":"2024-09-18T14:29:20.284127+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-18T14:29:32.300 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:32.300 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:32.637 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:32.641 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:32.655 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":723,"stamp":"2024-09-18T14:29:32.287294+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":14610,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":608112,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135568,"statfs":{"total":193273528320,"available":192650821632,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3034,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1517,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003207"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":16643,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:31.415899+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:31.415899+0000","last_peered":"2024-09-18T14:29:31.415899+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:31.415899+0000","last_undegraded":"2024-09-18T14:29:31.415899+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1016,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4023},{"shards":"2,3","objects":508}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":14610,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051640,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.373},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.64300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705947,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301976,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069864,"statfs":{"total":96636764160,"available":96327540736,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.33000000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:35.660 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:35.964 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:35.977 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":726,"stamp":"2024-09-18T14:29:35.489961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":14610,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612272,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131408,"statfs":{"total":193273528320,"available":192646561792,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2184,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1092,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.204410"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":16643,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:31.415899+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:31.415899+0000","last_peered":"2024-09-18T14:29:31.415899+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:31.415899+0000","last_undegraded":"2024-09-18T14:29:31.415899+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1016,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4023},{"shards":"2,3","objects":508}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":14610,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051641,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.373},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.64300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.34300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:35.978 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:36.298 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:36.300 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:36.311 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2024-09-18T14:29:36.288424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":14610,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612272,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131408,"statfs":{"total":193273528320,"available":192646561792,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1222,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002264"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":16643,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:31.415899+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:31.415899+0000","last_peered":"2024-09-18T14:29:31.415899+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:31.415899+0000","last_undegraded":"2024-09-18T14:29:31.415899+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1016,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4023},{"shards":"2,3","objects":508}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":14610,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051641,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.373},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.64300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.34300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:36.312 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:36.312 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4638, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094278+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:28:36.094278+0000', 'last_peered': '2024-09-18T14:28:36.094278+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:28:36.094278+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:36.312 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:36.312 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 4607, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.089025+0000', 'last_change': '2024-09-18T14:24:59.585939+0000', 'last_active': '2024-09-18T14:28:36.089025+0000', 'last_peered': '2024-09-18T14:28:36.089025+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:28:36.089025+0000', 'last_undegraded': '2024-09-18T14:24:59.535248+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4548, 'num_objects_misplaced': 4548, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4548}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:36.312 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:36.312 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4504, 'reported_epoch': 38, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:28:36.094272+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:28:36.094272+0000', 'last_peered': '2024-09-18T14:28:36.094272+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:28:36.094272+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:36.313 INFO:tasks.osd_recovery.ceph_manager:PG 1.0 is not active+clean 2024-09-18T14:29:36.313 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.0', 'version': "17'4531", 'reported_seq': 16643, 'reported_epoch': 39, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:31.415899+0000', 'last_change': '2024-09-18T14:28:35.828585+0000', 'last_active': '2024-09-18T14:29:31.415899+0000', 'last_peered': '2024-09-18T14:29:31.415899+0000', 'last_clean': '2024-09-18T14:23:59.311146+0000', 'last_became_active': '2024-09-18T14:24:48.462382+0000', 'last_became_peered': '2024-09-18T14:24:48.462382+0000', 'last_unstale': '2024-09-18T14:29:31.415899+0000', 'last_undegraded': '2024-09-18T14:29:31.415899+0000', 'last_fullsized': '2024-09-18T14:24:48.308994+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4531, 'log_dups_size': 0, 'ondisk_log_size': 4531, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 56991, 'num_objects': 4531, 'num_object_clones': 0, 'num_object_copies': 9062, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1016, 'num_objects_unfound': 0, 'num_objects_dirty': 4531, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4531, 'num_write_kb': 4579, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4023, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 4023}, {'shards': '2,3', 'objects': 508}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:36.313 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:36.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:36.711 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 13594/72472 objects degraded (18.758%), 3 pgs degraded, 4 pgs undersized","count":7},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1387,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":4},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":3},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":626966528,"bytes_avail":192646561792,"bytes_total":193273528320,"degraded_objects":13594,"degraded_total":72472,"degraded_ratio":0.18757589137873937,"misplaced_objects":14610,"misplaced_total":72472,"misplaced_ratio":0.20159509879677667,"recovering_objects_per_sec":76,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":611,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:36.726 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:36.726 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:37.021 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:37.024 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:37.037 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":727,"stamp":"2024-09-18T14:29:36.288424+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":14610,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612272,"kb_used_data":928,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131408,"statfs":{"total":193273528320,"available":192646561792,"internally_reserved":0,"allocated":950272,"data_stored":475219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1222,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":611,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002264"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4638,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094278+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:28:36.094278+0000","last_peered":"2024-09-18T14:28:36.094278+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:28:36.094278+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4607,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.089025+0000","last_change":"2024-09-18T14:24:59.585939+0000","last_active":"2024-09-18T14:28:36.089025+0000","last_peered":"2024-09-18T14:28:36.089025+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:28:36.089025+0000","last_undegraded":"2024-09-18T14:24:59.535248+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4548,"num_objects_misplaced":4548,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4548}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4504,"reported_epoch":38,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:28:36.094272+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:28:36.094272+0000","last_peered":"2024-09-18T14:28:36.094272+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:28:36.094272+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4565,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088449+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:28:36.088449+0000","last_peered":"2024-09-18T14:28:36.088449+0000","last_clean":"2024-09-18T14:28:36.088449+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:28:36.088449+0000","last_undegraded":"2024-09-18T14:28:36.088449+0000","last_fullsized":"2024-09-18T14:28:36.088449+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4528,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.090893+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:28:36.090893+0000","last_peered":"2024-09-18T14:28:36.090893+0000","last_clean":"2024-09-18T14:28:36.090893+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:28:36.090893+0000","last_undegraded":"2024-09-18T14:28:36.090893+0000","last_fullsized":"2024-09-18T14:28:36.090893+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4678,"reported_epoch":38,"state":"active+clean","last_fresh":"2024-09-18T14:28:36.088073+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:28:36.088073+0000","last_peered":"2024-09-18T14:28:36.088073+0000","last_clean":"2024-09-18T14:28:36.088073+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:28:36.088073+0000","last_undegraded":"2024-09-18T14:28:36.088073+0000","last_fullsized":"2024-09-18T14:28:36.088073+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4548,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-18T14:28:37.098807+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:28:37.098807+0000","last_peered":"2024-09-18T14:28:37.098807+0000","last_clean":"2024-09-18T14:28:37.098807+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:28:37.098807+0000","last_undegraded":"2024-09-18T14:28:37.098807+0000","last_fullsized":"2024-09-18T14:28:37.098807+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":16643,"reported_epoch":39,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:31.415899+0000","last_change":"2024-09-18T14:28:35.828585+0000","last_active":"2024-09-18T14:29:31.415899+0000","last_peered":"2024-09-18T14:29:31.415899+0000","last_clean":"2024-09-18T14:23:59.311146+0000","last_became_active":"2024-09-18T14:24:48.462382+0000","last_became_peered":"2024-09-18T14:24:48.462382+0000","last_unstale":"2024-09-18T14:29:31.415899+0000","last_undegraded":"2024-09-18T14:29:31.415899+0000","last_fullsized":"2024-09-18T14:24:48.308994+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T17:51:12.297074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1016,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4023,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4023},{"shards":"2,3","objects":508}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13594,"num_objects_misplaced":14610,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21075,"num_bytes_recovered":118082,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":942080,"data_stored":734620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":12,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051641,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182458,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.373},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.64300000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68400000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705948,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306136,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065704,"statfs":{"total":96636764160,"available":96323280896,"internally_reserved":0,"allocated":475136,"data_stored":237480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.53100000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.34300000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.57099999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":118063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:40.039 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:40.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:40.352 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:40.362 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2024-09-18T14:29:40.289509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17798,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21755,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":3188,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002214"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":5126,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:37.505410+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:37.505410+0000","last_peered":"2024-09-18T14:29:37.505410+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:37.505410+0000","last_undegraded":"2024-09-18T14:29:37.505410+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8752,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4376},{"shards":"3","objects":172}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17798,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21755,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051642,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705949,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.747}]}]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.71799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:40.363 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:40.675 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:40.679 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:40.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2024-09-18T14:29:40.289509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17798,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21755,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":3188,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002214"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":5126,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:37.505410+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:37.505410+0000","last_peered":"2024-09-18T14:29:37.505410+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:37.505410+0000","last_undegraded":"2024-09-18T14:29:37.505410+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8752,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4376},{"shards":"3","objects":172}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17798,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21755,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051642,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705949,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.747}]}]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.71799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:40.692 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:40.692 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:40.692 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:40.692 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 5126, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:37.505410+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:29:37.505410+0000', 'last_peered': '2024-09-18T14:29:37.505410+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:29:37.505410+0000', 'last_undegraded': '2024-09-18T14:29:37.505410+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8752, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 172, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4376}, {'shards': '3', 'objects': 172}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:40.692 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:40.692 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:40.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:41.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:41.059 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1391,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":627130368,"bytes_avail":192646397952,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":17798,"misplaced_total":72472,"misplaced_ratio":0.24558450160061818,"recovering_objects_per_sec":84,"recovering_bytes_per_sec":6556,"recovering_keys_per_sec":0,"num_objects_recovered":680,"num_bytes_recovered":52464,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:41.076 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:41.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:41.376 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:41.379 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:41.389 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":729,"stamp":"2024-09-18T14:29:40.289509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17798,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21755,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":3188,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002214"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":5126,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:37.505410+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:37.505410+0000","last_peered":"2024-09-18T14:29:37.505410+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:37.505410+0000","last_undegraded":"2024-09-18T14:29:37.505410+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8752,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":172,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4376},{"shards":"3","objects":172}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17798,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21755,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051642,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705949,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.63700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.51900000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.747}]}]},{"osd":0,"up_from":9,"seq":38654705949,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.71799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.67700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:44.391 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:44.720 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:44.723 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:44.733 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":731,"stamp":"2024-09-18T14:29:44.290454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22017,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":2664,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.813499"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":5913,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:42.495534+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:42.495534+0000","last_peered":"2024-09-18T14:29:42.495534+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:42.495534+0000","last_undegraded":"2024-09-18T14:29:42.495534+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8228,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4114},{"shards":"3","objects":434}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22017,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051643,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705950,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.65400000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.437},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:44.734 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:45.028 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:45.032 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:45.044 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":731,"stamp":"2024-09-18T14:29:44.290454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22017,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":2664,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.813499"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":5913,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:42.495534+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:42.495534+0000","last_peered":"2024-09-18T14:29:42.495534+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:42.495534+0000","last_undegraded":"2024-09-18T14:29:42.495534+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8228,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4114},{"shards":"3","objects":434}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22017,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051643,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705950,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.65400000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.437},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:45.045 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:45.045 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:45.045 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:45.045 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 5913, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:42.495534+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:29:42.495534+0000', 'last_peered': '2024-09-18T14:29:42.495534+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:29:42.495534+0000', 'last_undegraded': '2024-09-18T14:29:42.495534+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8228, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 434, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 4114}, {'shards': '3', 'objects': 434}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:45.046 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:45.046 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:45.046 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:45.408 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:45.409 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1396,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635650048,"bytes_avail":192637878272,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":17274,"misplaced_total":72472,"misplaced_ratio":0.23835412297163042,"recovering_objects_per_sec":95,"recovering_bytes_per_sec":5346,"recovering_keys_per_sec":0,"num_objects_recovered":942,"num_bytes_recovered":52464,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:45.422 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:45.422 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:45.709 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:45.712 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:45.727 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":731,"stamp":"2024-09-18T14:29:44.290454+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22017,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":2664,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":942,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.813499"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":5913,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:42.495534+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:42.495534+0000","last_peered":"2024-09-18T14:29:42.495534+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:42.495534+0000","last_undegraded":"2024-09-18T14:29:42.495534+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8228,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":434,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4114},{"shards":"3","objects":434}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":17274,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22017,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051643,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182460,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705950,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.439}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.40100000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.65400000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705950,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.39800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59099999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.437},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:48.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:49.048 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:49.052 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:49.062 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2024-09-18T14:29:48.291680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":2122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1213,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003256"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":6725,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:47.502570+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:47.502570+0000","last_peered":"2024-09-18T14:29:47.502570+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:47.502570+0000","last_undegraded":"2024-09-18T14:29:47.502570+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7686,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3843},{"shards":"3","objects":705}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051644,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182461,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.66900000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.63300000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.36899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.438},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:49.063 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:49.380 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:49.382 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:49.393 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2024-09-18T14:29:48.291680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":2122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1213,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003256"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":6725,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:47.502570+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:47.502570+0000","last_peered":"2024-09-18T14:29:47.502570+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:47.502570+0000","last_undegraded":"2024-09-18T14:29:47.502570+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7686,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3843},{"shards":"3","objects":705}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051644,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182461,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.66900000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.63300000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.36899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.438},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:49.394 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:49.394 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:49.394 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:49.394 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 6725, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:47.502570+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:29:47.502570+0000', 'last_peered': '2024-09-18T14:29:47.502570+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:29:47.502570+0000', 'last_undegraded': '2024-09-18T14:29:47.502570+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7686, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 705, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3843}, {'shards': '3', 'objects': 705}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:49.394 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:49.395 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:49.395 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:49.768 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:49.768 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1400,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635650048,"bytes_avail":192637878272,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":16732,"misplaced_total":72472,"misplaced_ratio":0.23087537255767746,"recovering_objects_per_sec":101,"recovering_bytes_per_sec":4370,"recovering_keys_per_sec":0,"num_objects_recovered":1213,"num_bytes_recovered":52464,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:49.782 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:49.782 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:50.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:50.102 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:50.113 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":733,"stamp":"2024-09-18T14:29:48.291680+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4548,"num_objects_misplaced":2122,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1213,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003256"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":6725,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:47.502570+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:47.502570+0000","last_peered":"2024-09-18T14:29:47.502570+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:47.502570+0000","last_undegraded":"2024-09-18T14:29:47.502570+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7686,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3843},{"shards":"3","objects":705}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051644,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182461,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.66900000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:28:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.502,"15min":0.48099999999999998},"min":{"1min":0.34200000000000003,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.75,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":1.589,"15min":2.6070000000000002},"min":{"1min":0.40999999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.81899999999999995,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.58699999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.63300000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.36899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.438},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:53.115 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:53.425 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:53.428 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:53.439 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2024-09-18T14:29:52.293042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1066,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003533"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":6725,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:47.502570+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:47.502570+0000","last_peered":"2024-09-18T14:29:47.502570+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:47.502570+0000","last_undegraded":"2024-09-18T14:29:47.502570+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7686,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3843},{"shards":"3","objects":705}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051644,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.84599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.499},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81399999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.36899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.438},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:53.440 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:53.749 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:53.752 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:53.763 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":735,"stamp":"2024-09-18T14:29:52.293042+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1066,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003533"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":6725,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:47.502570+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:47.502570+0000","last_peered":"2024-09-18T14:29:47.502570+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:47.502570+0000","last_undegraded":"2024-09-18T14:29:47.502570+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7686,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":705,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3843},{"shards":"3","objects":705}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16732,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22288,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051644,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.84599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.499},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81399999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705951,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.51500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.36899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.438},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:53.764 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:53.764 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:53.764 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:53.764 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 6725, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:47.502570+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:29:47.502570+0000', 'last_peered': '2024-09-18T14:29:47.502570+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:29:47.502570+0000', 'last_undegraded': '2024-09-18T14:29:47.502570+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7686, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 705, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3843}, {'shards': '3', 'objects': 705}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:53.764 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:53.764 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:53.765 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:54.143 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1405,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635650048,"bytes_avail":192637878272,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":16732,"misplaced_total":72472,"misplaced_ratio":0.23087537255767746,"recovering_objects_per_sec":44,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":533,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:54.156 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:54.156 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:54.452 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:54.455 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:54.465 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":736,"stamp":"2024-09-18T14:29:54.293637+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16164,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22572,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1634,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":817,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003625"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":7577,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:52.515853+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:52.515853+0000","last_peered":"2024-09-18T14:29:52.515853+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:52.515853+0000","last_undegraded":"2024-09-18T14:29:52.515853+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7118,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3559},{"shards":"3","objects":989}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16164,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22572,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051645,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.84599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.499},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81399999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:57.467 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:57.765 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:57.845 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:57.845 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2024-09-18T14:29:56.294290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16164,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22572,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1110,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003836"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":7577,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:52.515853+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:52.515853+0000","last_peered":"2024-09-18T14:29:52.515853+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:52.515853+0000","last_undegraded":"2024-09-18T14:29:52.515853+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7118,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3559},{"shards":"3","objects":989}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16164,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22572,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051645,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.84599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.499},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81399999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:57.846 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:58.151 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:58.154 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:58.165 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":737,"stamp":"2024-09-18T14:29:56.294290+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16164,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22572,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1110,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":555,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003836"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":7577,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:52.515853+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:52.515853+0000","last_peered":"2024-09-18T14:29:52.515853+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:52.515853+0000","last_undegraded":"2024-09-18T14:29:52.515853+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7118,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":989,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3559},{"shards":"3","objects":989}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":16164,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22572,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051645,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182462,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.84599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.499},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.71599999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81399999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705952,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.67600000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.52200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:29:58.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:29:58.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:58.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:29:58.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 7577, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:52.515853+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:29:52.515853+0000', 'last_peered': '2024-09-18T14:29:52.515853+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:29:52.515853+0000', 'last_undegraded': '2024-09-18T14:29:52.515853+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7118, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 989, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3559}, {'shards': '3', 'objects': 989}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:29:58.166 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:29:58.166 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:29:58.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:29:58.525 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:58.525 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1409,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635650048,"bytes_avail":192637878272,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":16164,"misplaced_total":72472,"misplaced_ratio":0.2230378628987747,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":555,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:29:58.542 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:29:58.542 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:29:58.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:29:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:29:58.880 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":738,"stamp":"2024-09-18T14:29:58.294706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":15560,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22874,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1714,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":857,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003633"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":8483,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:57.519687+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:57.519687+0000","last_peered":"2024-09-18T14:29:57.519687+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:57.519687+0000","last_undegraded":"2024-09-18T14:29:57.519687+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6514,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1291,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3257},{"shards":"3","objects":1291}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":15560,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22874,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051646,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705953,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72199999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.56000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:01.882 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:02.200 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:02.204 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:02.215 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":739,"stamp":"2024-09-18T14:30:00.295100+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":15560,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22874,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620752,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122928,"statfs":{"total":193273528320,"available":192637878272,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1172,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003420"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":8483,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:57.519687+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:57.519687+0000","last_peered":"2024-09-18T14:29:57.519687+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:57.519687+0000","last_undegraded":"2024-09-18T14:29:57.519687+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6514,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1291,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3257},{"shards":"3","objects":1291}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":15560,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22874,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051646,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705953,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.72199999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.56000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:02.217 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:02.509 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:02.511 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:02.522 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2024-09-18T14:30:02.295616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":15560,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22874,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624912,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188118768,"statfs":{"total":193273528320,"available":192633618432,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1172,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":8483,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:57.519687+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:57.519687+0000","last_peered":"2024-09-18T14:29:57.519687+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:57.519687+0000","last_undegraded":"2024-09-18T14:29:57.519687+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6514,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1291,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3257},{"shards":"3","objects":1291}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":15560,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22874,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051646,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314536,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057304,"statfs":{"total":96636764160,"available":96314679296,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.442},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.56000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:02.523 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:02.523 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:02.523 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:02.523 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 8483, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:29:57.519687+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:29:57.519687+0000', 'last_peered': '2024-09-18T14:29:57.519687+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:29:57.519687+0000', 'last_undegraded': '2024-09-18T14:29:57.519687+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6514, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1291, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 3257}, {'shards': '3', 'objects': 1291}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:02.523 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:02.523 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:02.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:02.909 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:02.909 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1413,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":639909888,"bytes_avail":192633618432,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":15560,"misplaced_total":72472,"misplaced_ratio":0.21470360966994145,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":586,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:02.925 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:02.925 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:03.227 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:03.231 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:03.245 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":740,"stamp":"2024-09-18T14:30:02.295616+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":15560,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22874,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":624912,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":623804,"kb_avail":188118768,"statfs":{"total":193273528320,"available":192633618432,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":638775984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1172,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":586,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003209"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":8483,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:29:57.519687+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:29:57.519687+0000","last_peered":"2024-09-18T14:29:57.519687+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:29:57.519687+0000","last_undegraded":"2024-09-18T14:29:57.519687+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6514,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1291,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":3257},{"shards":"3","objects":1291}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":15560,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22874,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051646,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314536,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057304,"statfs":{"total":96636764160,"available":96314679296,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.442},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705953,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.56000000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:28:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53100000000000003,"15min":0.502},"min":{"1min":0.41499999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.73899999999999999,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.57299999999999995,"15min":0.54000000000000004},"min":{"1min":0.35799999999999998,"5min":0.247,"15min":0.247},"max":{"1min":0.88200000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.59899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:06.247 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:06.560 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:06.563 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:06.576 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2024-09-18T14:30:06.296597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14914,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1250,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002960"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":9452,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:02.519759+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:02.519759+0000","last_peered":"2024-09-18T14:30:02.519759+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:02.519759+0000","last_undegraded":"2024-09-18T14:30:02.519759+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5868,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2934},{"shards":"3","objects":1614}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14914,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051647,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314536,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057304,"statfs":{"total":96636764160,"available":96314679296,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.442},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:06.577 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:06.897 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:06.900 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:06.911 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2024-09-18T14:30:06.296597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14914,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1250,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002960"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":9452,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:02.519759+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:02.519759+0000","last_peered":"2024-09-18T14:30:02.519759+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:02.519759+0000","last_undegraded":"2024-09-18T14:30:02.519759+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5868,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2934},{"shards":"3","objects":1614}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14914,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051647,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314536,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057304,"statfs":{"total":96636764160,"available":96314679296,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.442},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:06.912 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:06.912 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:06.912 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:06.912 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 9452, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:02.519759+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:30:02.519759+0000', 'last_peered': '2024-09-18T14:30:02.519759+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:30:02.519759+0000', 'last_undegraded': '2024-09-18T14:30:02.519759+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5868, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1614, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2934}, {'shards': '3', 'objects': 1614}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:06.913 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:06.913 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:06.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:07.289 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:07.290 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1418,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":627130368,"bytes_avail":192646397952,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":14914,"misplaced_total":72472,"misplaced_ratio":0.20578982227618942,"recovering_objects_per_sec":52,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":625,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:07.304 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:07.304 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:07.600 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:07.604 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:07.617 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":742,"stamp":"2024-09-18T14:30:06.296597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14914,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1250,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":625,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002960"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":9452,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:02.519759+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:02.519759+0000","last_peered":"2024-09-18T14:30:02.519759+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:02.519759+0000","last_undegraded":"2024-09-18T14:30:02.519759+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5868,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1614,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2934},{"shards":"3","objects":1614}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14914,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23197,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051647,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182464,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314536,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057304,"statfs":{"total":96636764160,"available":96314679296,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.497}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.442},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.41899999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.68500000000000005}]}]},{"osd":0,"up_from":9,"seq":38654705954,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:10.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:10.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:10.952 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2024-09-18T14:30:10.297739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14232,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23538,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595792,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147888,"statfs":{"total":193273528320,"available":192663437312,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1328,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003033"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":10475,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:07.522471+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:07.522471+0000","last_peered":"2024-09-18T14:30:07.522471+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:07.522471+0000","last_undegraded":"2024-09-18T14:30:07.522471+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5186,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1955,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2593},{"shards":"3","objects":1955}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14232,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23538,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051648,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.438},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59999999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.86099999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.42299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:10.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:11.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:11.264 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:11.276 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2024-09-18T14:30:10.297739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14232,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23538,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595792,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147888,"statfs":{"total":193273528320,"available":192663437312,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1328,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003033"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":10475,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:07.522471+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:07.522471+0000","last_peered":"2024-09-18T14:30:07.522471+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:07.522471+0000","last_undegraded":"2024-09-18T14:30:07.522471+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5186,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1955,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2593},{"shards":"3","objects":1955}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14232,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23538,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051648,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.438},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59999999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.86099999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.42299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:11.277 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:11.277 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:11.278 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:11.278 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 10475, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:07.522471+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:30:07.522471+0000', 'last_peered': '2024-09-18T14:30:07.522471+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:30:07.522471+0000', 'last_undegraded': '2024-09-18T14:30:07.522471+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5186, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1955, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2593}, {'shards': '3', 'objects': 1955}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:11.278 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:11.278 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:11.278 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:11.636 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:11.637 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1422,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":610091008,"bytes_avail":192663437312,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":14232,"misplaced_total":72472,"misplaced_ratio":0.19637929131250689,"recovering_objects_per_sec":55,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":664,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:11.651 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:11.651 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:11.975 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:11.978 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:11.991 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":744,"stamp":"2024-09-18T14:30:10.297739+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14232,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23538,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595792,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147888,"statfs":{"total":193273528320,"available":192663437312,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1328,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":664,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003033"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":10475,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:07.522471+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:07.522471+0000","last_peered":"2024-09-18T14:30:07.522471+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:07.522471+0000","last_undegraded":"2024-09-18T14:30:07.522471+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5186,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1955,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2593},{"shards":"3","objects":1955}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":14232,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23538,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051648,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182465,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.438},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59999999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.56000000000000005,"15min":0.54000000000000004},"min":{"1min":0.39000000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.68799999999999994,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.66100000000000003,"15min":0.63200000000000001},"min":{"1min":0.48799999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.96399999999999997,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.86099999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705955,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.42299999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.56399999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:29:09 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.38,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79700000000000004,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.623,"15min":0.61899999999999999},"min":{"1min":0.436,"5min":0.317,"15min":0.317},"max":{"1min":0.93000000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.69299999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:14.993 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:15.318 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:15.320 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:15.330 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2024-09-18T14:30:14.298844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":13494,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23907,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595792,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147888,"statfs":{"total":193273528320,"available":192663437312,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2066,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003227"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":11582,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:12.525401+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:12.525401+0000","last_peered":"2024-09-18T14:30:12.525401+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:12.525401+0000","last_undegraded":"2024-09-18T14:30:12.525401+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4448,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2224},{"shards":"3","objects":2324}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":13494,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23907,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051649,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182466,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705956,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:15.331 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:15.651 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:15.654 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:15.665 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":746,"stamp":"2024-09-18T14:30:14.298844+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":13494,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23907,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595792,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147888,"statfs":{"total":193273528320,"available":192663437312,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2066,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1033,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003227"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":11582,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:12.525401+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:12.525401+0000","last_peered":"2024-09-18T14:30:12.525401+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:12.525401+0000","last_undegraded":"2024-09-18T14:30:12.525401+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4448,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2224},{"shards":"3","objects":2324}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":13494,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23907,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051649,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182466,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705956,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:15.666 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:15.666 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:15.666 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:15.666 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 11582, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:12.525401+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:30:12.525401+0000', 'last_peered': '2024-09-18T14:30:12.525401+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:30:12.525401+0000', 'last_undegraded': '2024-09-18T14:30:12.525401+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4448, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2324, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 2224}, {'shards': '3', 'objects': 2324}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:15.666 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:15.666 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:15.667 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:16.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:16.044 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1426,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":610091008,"bytes_avail":192663437312,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":13494,"misplaced_total":72472,"misplaced_ratio":0.18619604812893256,"recovering_objects_per_sec":86,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1033,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:16.061 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:16.062 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:16.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:16.365 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:16.375 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":747,"stamp":"2024-09-18T14:30:16.299573+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":13494,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23907,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595792,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147888,"statfs":{"total":193273528320,"available":192663437312,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1420,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":710,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003531"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":11582,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:12.525401+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:12.525401+0000","last_peered":"2024-09-18T14:30:12.525401+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:12.525401+0000","last_undegraded":"2024-09-18T14:30:12.525401+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4448,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2324,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":2224},{"shards":"3","objects":2324}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":13494,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23907,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051649,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182466,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705956,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.55600000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705956,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297896,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073944,"statfs":{"total":96636764160,"available":96331718656,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.46200000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.56299999999999994}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.61799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:19.377 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:19.698 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:19.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2024-09-18T14:30:18.300043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":12694,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24307,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604112,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139568,"statfs":{"total":193273528320,"available":192654917632,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2220,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":12782,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:17.529155+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:17.529155+0000","last_peered":"2024-09-18T14:30:17.529155+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:17.529155+0000","last_undegraded":"2024-09-18T14:30:17.529155+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3648,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2724,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1824},{"shards":"3","objects":2724}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":12694,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24307,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051650,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182467,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705957,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.54200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.378},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.504},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:19.716 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:20.012 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:20.016 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:20.031 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":748,"stamp":"2024-09-18T14:30:18.300043+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":12694,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24307,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604112,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139568,"statfs":{"total":193273528320,"available":192654917632,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2220,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1110,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003445"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":12782,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:17.529155+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:17.529155+0000","last_peered":"2024-09-18T14:30:17.529155+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:17.529155+0000","last_undegraded":"2024-09-18T14:30:17.529155+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3648,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2724,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1824},{"shards":"3","objects":2724}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":12694,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24307,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051650,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182467,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705957,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.54200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.378},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.504},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:20.032 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:20.032 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:20.033 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:20.033 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 12782, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:17.529155+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:30:17.529155+0000', 'last_peered': '2024-09-18T14:30:17.529155+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:30:17.529155+0000', 'last_undegraded': '2024-09-18T14:30:17.529155+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3648, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2724, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1824}, {'shards': '3', 'objects': 2724}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:20.033 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:20.033 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:20.033 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:20.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:20.500 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1431,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618610688,"bytes_avail":192654917632,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":12694,"misplaced_total":72472,"misplaced_ratio":0.17515730213047795,"recovering_objects_per_sec":92,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1110,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:20.518 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:20.518 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:20.847 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:20.850 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:20.863 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":749,"stamp":"2024-09-18T14:30:20.300535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":12694,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24307,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604112,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139568,"statfs":{"total":193273528320,"available":192654917632,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1538,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003466"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":12782,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:17.529155+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:17.529155+0000","last_peered":"2024-09-18T14:30:17.529155+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:17.529155+0000","last_undegraded":"2024-09-18T14:30:17.529155+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3648,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2724,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1824},{"shards":"3","objects":2724}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":12694,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24307,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051650,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182467,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705957,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45200000000000001,"15min":0.53900000000000003},"min":{"1min":0.30399999999999999,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.73699999999999999,"5min":0.97299999999999998,"15min":6.758},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.625,"5min":0.56200000000000006,"15min":0.60899999999999999},"min":{"1min":0.377,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.89200000000000002,"5min":1.6399999999999999,"15min":6.7859999999999996},"last":0.54000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61299999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.54200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.378},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.504},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:23.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:24.168 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:24.184 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2024-09-18T14:30:22.301041+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":12694,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24307,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604112,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139568,"statfs":{"total":193273528320,"available":192654917632,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1538,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":769,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003302"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":12782,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:17.529155+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:17.529155+0000","last_peered":"2024-09-18T14:30:17.529155+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:17.529155+0000","last_undegraded":"2024-09-18T14:30:17.529155+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3648,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2724,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1824},{"shards":"3","objects":2724}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":12694,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24307,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051650,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182468,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705958,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705957,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:29:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.45700000000000002,"15min":0.52500000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":1.3959999999999999,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.57199999999999995,"15min":0.64300000000000002},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":1.542,"5min":1.542,"15min":1.542},"last":0.54200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.378},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.504},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.72799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:24.185 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:24.510 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:24.523 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":751,"stamp":"2024-09-18T14:30:24.301547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":11794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24757,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604112,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139568,"statfs":{"total":193273528320,"available":192654917632,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2438,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003203"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":14132,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:22.532788+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:22.532788+0000","last_peered":"2024-09-18T14:30:22.532788+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:22.532788+0000","last_undegraded":"2024-09-18T14:30:22.532788+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2748,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1374},{"shards":"3","objects":3174}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":11794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24757,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051651,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182468,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705958,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.495},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.371}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.77600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:24.524 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:24.525 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:24.525 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:24.525 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 14132, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:22.532788+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:30:22.532788+0000', 'last_peered': '2024-09-18T14:30:22.532788+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:30:22.532788+0000', 'last_undegraded': '2024-09-18T14:30:22.532788+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2748, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3174, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 1374}, {'shards': '3', 'objects': 3174}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:24.525 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:24.525 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:24.526 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:24.907 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:24.907 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1435,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618610688,"bytes_avail":192654917632,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":11794,"misplaced_total":72472,"misplaced_ratio":0.16273871288221659,"recovering_objects_per_sec":101,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1219,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:24.921 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:24.921 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:25.233 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:25.236 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:25.250 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":751,"stamp":"2024-09-18T14:30:24.301547+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":11794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24757,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604112,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139568,"statfs":{"total":193273528320,"available":192654917632,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2438,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1219,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003203"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":14132,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:22.532788+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:22.532788+0000","last_peered":"2024-09-18T14:30:22.532788+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:22.532788+0000","last_undegraded":"2024-09-18T14:30:22.532788+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2748,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3174,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":1374},{"shards":"3","objects":3174}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":11794,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24757,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051651,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182468,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705958,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.63800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.47299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67000000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705958,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302056,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069784,"statfs":{"total":96636764160,"available":96327458816,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.61899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.495},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.371}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.77600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:28.252 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:28.598 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:28.601 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:28.611 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2024-09-18T14:30:28.302525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2772,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":15740,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:27.530551+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:27.530551+0000","last_peered":"2024-09-18T14:30:27.530551+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:27.530551+0000","last_undegraded":"2024-09-18T14:30:27.530551+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1676,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3710,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":838},{"shards":"3","objects":3710}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051652,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182469,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:28.612 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:28.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:28.940 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:28.953 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2024-09-18T14:30:28.302525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2772,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":15740,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:27.530551+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:27.530551+0000","last_peered":"2024-09-18T14:30:27.530551+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:27.530551+0000","last_undegraded":"2024-09-18T14:30:27.530551+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1676,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3710,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":838},{"shards":"3","objects":3710}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051652,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182469,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:28.954 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:28.954 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:28.954 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:28.954 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 15740, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:27.530551+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:30:27.530551+0000', 'last_peered': '2024-09-18T14:30:27.530551+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:30:27.530551+0000', 'last_undegraded': '2024-09-18T14:30:27.530551+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1676, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3710, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 838}, {'shards': '3', 'objects': 3710}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:28.954 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:28.954 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:28.955 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:29.352 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:29.352 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1440,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":41,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":3},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":627130368,"bytes_avail":192646397952,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":10722,"misplaced_total":72472,"misplaced_ratio":0.14794679324428744,"recovering_objects_per_sec":115,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1386,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:29.368 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:29.368 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:29.679 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:29.681 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:29.693 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":753,"stamp":"2024-09-18T14:30:28.302525+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2772,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1386,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002951"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":15740,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:27.530551+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:27.530551+0000","last_peered":"2024-09-18T14:30:27.530551+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:27.530551+0000","last_undegraded":"2024-09-18T14:30:27.530551+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1676,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3710,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":838},{"shards":"3","objects":3710}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051652,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182469,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.67000000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.59499999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.54300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:32.695 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:33.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:33.019 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:33.031 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-09-18T14:30:32.303577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1972,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003042"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":15740,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:27.530551+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:27.530551+0000","last_peered":"2024-09-18T14:30:27.530551+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:27.530551+0000","last_undegraded":"2024-09-18T14:30:27.530551+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1676,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3710,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":838},{"shards":"3","objects":3710}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051652,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.437},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.56599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.623}]}]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:33.032 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:33.332 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:33.345 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-09-18T14:30:32.303577+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612432,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131248,"statfs":{"total":193273528320,"available":192646397952,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1972,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":986,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003042"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":15740,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:27.530551+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:27.530551+0000","last_peered":"2024-09-18T14:30:27.530551+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:27.530551+0000","last_undegraded":"2024-09-18T14:30:27.530551+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1676,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3710,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":838},{"shards":"3","objects":3710}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":10722,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25293,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1073152,"data_stored":839548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051652,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.437},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.56599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.623}]}]},{"osd":0,"up_from":9,"seq":38654705959,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.56399999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.60399999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:33.346 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:33.346 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:33.346 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:33.346 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 15740, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:27.530551+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:30:27.530551+0000', 'last_peered': '2024-09-18T14:30:27.530551+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:30:27.530551+0000', 'last_undegraded': '2024-09-18T14:30:27.530551+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 1676, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3710, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 838}, {'shards': '3', 'objects': 3710}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:33.346 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:33.346 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:33.347 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:33.719 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:33.719 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1444,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":42,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":627130368,"bytes_avail":192646397952,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":10722,"misplaced_total":72472,"misplaced_ratio":0.14794679324428744,"recovering_objects_per_sec":82,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":986,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:33.732 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:33.732 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:34.064 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:34.079 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":756,"stamp":"2024-09-18T14:30:33.700083+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":9148,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616592,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188127088,"statfs":{"total":193273528320,"available":192642138112,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3546,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1773,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.399041"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":18101,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:32.541301+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:32.541301+0000","last_peered":"2024-09-18T14:30:32.541301+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:32.541301+0000","last_undegraded":"2024-09-18T14:30:32.541301+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":102,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":51},{"shards":"3","objects":4497}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":9148,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051653,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.437},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.56599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.623}]}]},{"osd":0,"up_from":9,"seq":38654705960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.373},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.61399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.85599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:37.080 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:37.390 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:37.393 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:37.404 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2024-09-18T14:30:36.304644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":9148,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616592,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188127088,"statfs":{"total":193273528320,"available":192642138112,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1574,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002119"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":18101,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:32.541301+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:32.541301+0000","last_peered":"2024-09-18T14:30:32.541301+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:32.541301+0000","last_undegraded":"2024-09-18T14:30:32.541301+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":102,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":51},{"shards":"3","objects":4497}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":9148,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051653,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.437},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.56599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.623}]}]},{"osd":0,"up_from":9,"seq":38654705960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.373},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.61399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.85599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:37.405 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:37.681 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:37.694 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":759,"stamp":"2024-09-18T14:30:36.304644+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":9148,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616592,"kb_used_data":1088,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188127088,"statfs":{"total":193273528320,"available":192642138112,"internally_reserved":0,"allocated":1114112,"data_stored":593217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1574,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002119"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4640,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.485873+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:29:34.485873+0000","last_peered":"2024-09-18T14:29:34.485873+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:29:34.485873+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":18101,"reported_epoch":41,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:32.541301+0000","last_change":"2024-09-18T14:29:34.155549+0000","last_active":"2024-09-18T14:30:32.541301+0000","last_peered":"2024-09-18T14:30:32.541301+0000","last_clean":"2024-09-18T14:23:58.782621+0000","last_became_active":"2024-09-18T14:24:59.543684+0000","last_became_peered":"2024-09-18T14:24:59.543684+0000","last_unstale":"2024-09-18T14:30:32.541301+0000","last_undegraded":"2024-09-18T14:30:32.541301+0000","last_fullsized":"2024-09-18T14:24:59.421342+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":102,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4497,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[3],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":51},{"shards":"3","objects":4497}],"blocked_by":[],"up_primary":0,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4506,"reported_epoch":40,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:29:34.486444+0000","last_change":"2024-09-18T14:24:48.470413+0000","last_active":"2024-09-18T14:29:34.486444+0000","last_peered":"2024-09-18T14:29:34.486444+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:29:34.486444+0000","last_undegraded":"2024-09-18T14:24:48.458742+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4453,"num_objects_misplaced":4453,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4453}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4567,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484821+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:29:34.484821+0000","last_peered":"2024-09-18T14:29:34.484821+0000","last_clean":"2024-09-18T14:29:34.484821+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:29:34.484821+0000","last_undegraded":"2024-09-18T14:29:34.484821+0000","last_fullsized":"2024-09-18T14:29:34.484821+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4530,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.482067+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:29:34.482067+0000","last_peered":"2024-09-18T14:29:34.482067+0000","last_clean":"2024-09-18T14:29:34.482067+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:29:34.482067+0000","last_undegraded":"2024-09-18T14:29:34.482067+0000","last_fullsized":"2024-09-18T14:29:34.482067+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4680,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484838+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:29:34.484838+0000","last_peered":"2024-09-18T14:29:34.484838+0000","last_clean":"2024-09-18T14:29:34.484838+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:29:34.484838+0000","last_undegraded":"2024-09-18T14:29:34.484838+0000","last_fullsized":"2024-09-18T14:29:34.484838+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4550,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-18T14:29:34.484778+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:29:34.484778+0000","last_peered":"2024-09-18T14:29:34.484778+0000","last_clean":"2024-09-18T14:29:34.484778+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:29:34.484778+0000","last_undegraded":"2024-09-18T14:29:34.484778+0000","last_fullsized":"2024-09-18T14:29:34.484778+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4563,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-09-18T14:29:35.501274+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:29:35.501274+0000","last_peered":"2024-09-18T14:29:35.501274+0000","last_clean":"2024-09-18T14:29:35.501274+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:29:35.501274+0000","last_undegraded":"2024-09-18T14:29:35.501274+0000","last_fullsized":"2024-09-18T14:29:35.501274+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9046,"num_objects_misplaced":9148,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26080,"num_bytes_recovered":170546,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":13,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051653,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306216,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065624,"statfs":{"total":96636764160,"available":96323198976,"internally_reserved":0,"allocated":557056,"data_stored":296738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.437},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.56599999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.68400000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.623}]}]},{"osd":0,"up_from":9,"seq":38654705960,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310376,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061464,"statfs":{"total":96636764160,"available":96318939136,"internally_reserved":0,"allocated":557056,"data_stored":296479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.373},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.61399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.85599999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.56499999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":170527,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:37.695 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:37.695 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4640, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.485873+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:29:34.485873+0000', 'last_peered': '2024-09-18T14:29:34.485873+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:29:34.485873+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:37.695 INFO:tasks.osd_recovery.ceph_manager:PG 1.6 is not active+clean 2024-09-18T14:30:37.695 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.6', 'version': "17'4548", 'reported_seq': 18101, 'reported_epoch': 41, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:32.541301+0000', 'last_change': '2024-09-18T14:29:34.155549+0000', 'last_active': '2024-09-18T14:30:32.541301+0000', 'last_peered': '2024-09-18T14:30:32.541301+0000', 'last_clean': '2024-09-18T14:23:58.782621+0000', 'last_became_active': '2024-09-18T14:24:59.543684+0000', 'last_became_peered': '2024-09-18T14:24:59.543684+0000', 'last_unstale': '2024-09-18T14:30:32.541301+0000', 'last_undegraded': '2024-09-18T14:30:32.541301+0000', 'last_fullsized': '2024-09-18T14:24:59.421342+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4548, 'log_dups_size': 0, 'ondisk_log_size': 4548, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T21:35:19.617462+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000972099, 'stat_sum': {'num_bytes': 57008, 'num_objects': 4548, 'num_object_clones': 0, 'num_object_copies': 9096, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 102, 'num_objects_unfound': 0, 'num_objects_dirty': 4548, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4548, 'num_write_kb': 4596, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4497, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [3], 'avail_no_missing': ['3', '2'], 'object_location_counts': [{'shards': '2,3', 'objects': 51}, {'shards': '3', 'objects': 4497}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 3, 'purged_snaps': []} 2024-09-18T14:30:37.695 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:37.695 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 4506, 'reported_epoch': 40, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:29:34.486444+0000', 'last_change': '2024-09-18T14:24:48.470413+0000', 'last_active': '2024-09-18T14:29:34.486444+0000', 'last_peered': '2024-09-18T14:29:34.486444+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:29:34.486444+0000', 'last_undegraded': '2024-09-18T14:24:48.458742+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4453, 'num_objects_misplaced': 4453, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4453}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:37.696 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:38.056 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:38.056 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 9046/72472 objects degraded (12.482%), 2 pgs degraded, 3 pgs undersized","count":5},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1448,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":5},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":2},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":631390208,"bytes_avail":192642138112,"bytes_total":193273528320,"degraded_objects":9046,"degraded_total":72472,"degraded_ratio":0.12482062037752512,"misplaced_objects":9148,"misplaced_total":72472,"misplaced_ratio":0.12622806049232807,"recovering_objects_per_sec":98,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":787,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:38.070 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:38.071 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:38.384 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:38.388 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:38.400 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":760,"stamp":"2024-09-18T14:30:38.305451+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":13119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26321,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4453,"num_objects_misplaced":2397,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1028,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002329"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":5079,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:36.417552+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:36.417552+0000","last_peered":"2024-09-18T14:30:36.417552+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:36.417552+0000","last_undegraded":"2024-09-18T14:30:36.417552+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8526,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":190},{"shards":"2,3","objects":4263}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":13119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26321,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":931360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051654,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182471,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.39900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.42699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.90500000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.54900000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.627},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":236131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:41.402 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:41.725 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:41.740 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":761,"stamp":"2024-09-18T14:30:40.306063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":13119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26321,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4453,"num_objects_misplaced":2397,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1028,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002486"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":5079,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:36.417552+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:36.417552+0000","last_peered":"2024-09-18T14:30:36.417552+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:36.417552+0000","last_undegraded":"2024-09-18T14:30:36.417552+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8526,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":190},{"shards":"2,3","objects":4263}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":13119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26321,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":931360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051654,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182471,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.39900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.42699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.90500000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.54900000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.627},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":236131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:41.741 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:42.052 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:42.070 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":761,"stamp":"2024-09-18T14:30:40.306063+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":13119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26321,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4453,"num_objects_misplaced":2397,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1028,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002486"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":5079,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:36.417552+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:36.417552+0000","last_peered":"2024-09-18T14:30:36.417552+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:36.417552+0000","last_undegraded":"2024-09-18T14:30:36.417552+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8526,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":190,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":190},{"shards":"2,3","objects":4263}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":13119,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26321,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":931360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051654,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182471,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.39900000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.42699999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.90500000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.54900000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.627},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":236131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:42.070 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:42.071 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:42.071 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:42.071 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 5079, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:36.417552+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:30:36.417552+0000', 'last_peered': '2024-09-18T14:30:36.417552+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:30:36.417552+0000', 'last_undegraded': '2024-09-18T14:30:36.417552+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8526, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 190, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 190}, {'shards': '2,3', 'objects': 4263}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:42.071 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:42.440 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:42.440 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1453,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635813888,"bytes_avail":192637714432,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":13119,"misplaced_total":72472,"misplaced_ratio":0.18102163594215695,"recovering_objects_per_sec":128,"recovering_bytes_per_sec":6555,"recovering_keys_per_sec":0,"num_objects_recovered":1028,"num_bytes_recovered":52464,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:42.454 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:42.454 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:42.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:42.749 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:42.760 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":762,"stamp":"2024-09-18T14:30:42.306541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":12561,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26600,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4453,"num_objects_misplaced":3413,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.606458"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":5916,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:41.423100+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:41.423100+0000","last_peered":"2024-09-18T14:30:41.423100+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:41.423100+0000","last_undegraded":"2024-09-18T14:30:41.423100+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7968,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":469},{"shards":"2,3","objects":3984}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":12561,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26600,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":931360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051654,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182472,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705961,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.50700000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.54900000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.627},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.40500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":236131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:45.762 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:46.085 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:46.088 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:46.098 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":763,"stamp":"2024-09-18T14:30:44.306905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":12561,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26600,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4453,"num_objects_misplaced":3413,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002793"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":5916,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:41.423100+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:41.423100+0000","last_peered":"2024-09-18T14:30:41.423100+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:41.423100+0000","last_undegraded":"2024-09-18T14:30:41.423100+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7968,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":469},{"shards":"2,3","objects":3984}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":12561,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26600,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171456,"data_stored":918244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051655,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182472,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.58999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":223015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:46.099 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:46.398 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:46.403 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:46.419 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2024-09-18T14:30:46.307396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":12561,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26600,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4453,"num_objects_misplaced":3413,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.599613"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":5916,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:41.423100+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:41.423100+0000","last_peered":"2024-09-18T14:30:41.423100+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:41.423100+0000","last_undegraded":"2024-09-18T14:30:41.423100+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7968,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":469},{"shards":"2,3","objects":3984}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":12561,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26600,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171456,"data_stored":918244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051655,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182472,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.58999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":223015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:46.420 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:46.420 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:46.420 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:46.420 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 5916, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:41.423100+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:30:41.423100+0000', 'last_peered': '2024-09-18T14:30:41.423100+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:30:41.423100+0000', 'last_undegraded': '2024-09-18T14:30:41.423100+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7968, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 469, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 469}, {'shards': '2,3', 'objects': 3984}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:46.420 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:46.795 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:46.795 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1457,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635813888,"bytes_avail":192637714432,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":12561,"misplaced_total":72472,"misplaced_ratio":0.17332211060823488,"recovering_objects_per_sec":44,"recovering_bytes_per_sec":4522,"recovering_keys_per_sec":0,"num_objects_recovered":520,"num_bytes_recovered":52464,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:46.809 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:46.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:47.114 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:47.117 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:47.128 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":764,"stamp":"2024-09-18T14:30:46.307396+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":12561,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26600,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4453,"num_objects_misplaced":3413,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":520,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.599613"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":5916,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:41.423100+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:41.423100+0000","last_peered":"2024-09-18T14:30:41.423100+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:41.423100+0000","last_undegraded":"2024-09-18T14:30:41.423100+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7968,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":469},{"shards":"2,3","objects":3984}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":12561,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26600,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171456,"data_stored":918244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051655,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182472,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.45800000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.76300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61699999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705962,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.67200000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.58999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":223015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:50.130 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:50.439 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:50.442 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:50.454 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2024-09-18T14:30:50.308558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11987,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26887,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1132,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":566,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":6777,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:46.423955+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:46.423955+0000","last_peered":"2024-09-18T14:30:46.423955+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:46.423955+0000","last_undegraded":"2024-09-18T14:30:46.423955+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7394,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":756},{"shards":"2,3","objects":3697}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11987,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26887,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171456,"data_stored":918244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051656,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182473,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.876}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.497},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.78600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74099999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705963,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.53200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42999999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":223015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:50.455 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:50.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:50.776 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:50.786 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2024-09-18T14:30:50.308558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11987,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26887,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1132,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":566,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":6777,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:46.423955+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:46.423955+0000","last_peered":"2024-09-18T14:30:46.423955+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:46.423955+0000","last_undegraded":"2024-09-18T14:30:46.423955+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7394,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":756},{"shards":"2,3","objects":3697}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11987,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26887,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171456,"data_stored":918244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051656,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182473,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.876}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.497},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.78600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74099999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705963,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.53200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42999999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":223015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:50.787 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:50.787 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:50.787 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:50.787 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 6777, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:46.423955+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:30:46.423955+0000', 'last_peered': '2024-09-18T14:30:46.423955+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:30:46.423955+0000', 'last_undegraded': '2024-09-18T14:30:46.423955+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7394, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 756, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 756}, {'shards': '2,3', 'objects': 3697}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:50.787 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:51.182 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:51.183 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1462,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635813888,"bytes_avail":192637714432,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":11987,"misplaced_total":72472,"misplaced_ratio":0.16540181035434376,"recovering_objects_per_sec":47,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":566,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:51.201 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:51.201 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:51.528 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:51.532 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:51.543 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":766,"stamp":"2024-09-18T14:30:50.308558+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11987,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26887,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1132,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":566,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003107"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":6777,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:46.423955+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:46.423955+0000","last_peered":"2024-09-18T14:30:46.423955+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:46.423955+0000","last_undegraded":"2024-09-18T14:30:46.423955+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7394,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":756,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":756},{"shards":"2,3","objects":3697}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11987,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26887,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1171456,"data_stored":918244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051656,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182473,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705963,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.876}]},{"osd":2,"last update":"Wed Sep 18 14:29:51 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.51500000000000001,"15min":0.48499999999999999},"min":{"1min":0.34799999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.357,"5min":1.357,"15min":1.357},"last":0.497},{"interface":"front","average":{"1min":0.54300000000000004,"5min":1.075,"15min":2.4359999999999999},"min":{"1min":0.39700000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.80600000000000005,"5min":46.807000000000002,"15min":46.807000000000002},"last":0.78600000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74099999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705963,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.53200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42999999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.41299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":223015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:54.545 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:54.844 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:54.858 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":768,"stamp":"2024-09-18T14:30:54.309753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11395,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27183,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1166,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":7665,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:51.432887+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:51.432887+0000","last_peered":"2024-09-18T14:30:51.432887+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:51.432887+0000","last_undegraded":"2024-09-18T14:30:51.432887+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6802,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1052},{"shards":"2,3","objects":3401}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11395,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27183,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051657,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182474,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705964,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314616,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057224,"statfs":{"total":96636764160,"available":96314597376,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.55300000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53100000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705964,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.56799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:54.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:55.156 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:55.158 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:55.171 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":768,"stamp":"2024-09-18T14:30:54.309753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11395,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27183,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1166,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":7665,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:51.432887+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:51.432887+0000","last_peered":"2024-09-18T14:30:51.432887+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:51.432887+0000","last_undegraded":"2024-09-18T14:30:51.432887+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6802,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1052},{"shards":"2,3","objects":3401}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11395,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27183,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051657,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182474,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705964,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314616,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057224,"statfs":{"total":96636764160,"available":96314597376,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.55300000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53100000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705964,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.56799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:55.172 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:30:55.172 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:55.173 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:30:55.173 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 7665, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:30:51.432887+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:30:51.432887+0000', 'last_peered': '2024-09-18T14:30:51.432887+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:30:51.432887+0000', 'last_undegraded': '2024-09-18T14:30:51.432887+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6802, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1052, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1052}, {'shards': '2,3', 'objects': 3401}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:30:55.174 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:30:55.551 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:55.552 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1466,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":627294208,"bytes_avail":192646234112,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":11395,"misplaced_total":72472,"misplaced_ratio":0.15723313831548735,"recovering_objects_per_sec":48,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":583,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:30:55.565 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:30:55.565 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:30:55.884 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:30:55.887 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:30:55.902 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":768,"stamp":"2024-09-18T14:30:54.309753+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11395,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27183,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1166,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":583,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003212"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":7665,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:30:51.432887+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:30:51.432887+0000","last_peered":"2024-09-18T14:30:51.432887+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:30:51.432887+0000","last_undegraded":"2024-09-18T14:30:51.432887+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6802,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1052,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1052},{"shards":"2,3","objects":3401}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":11395,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27183,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051657,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182474,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705964,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314616,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057224,"statfs":{"total":96636764160,"available":96314597376,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.55300000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.58899999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53100000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705964,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.56799999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61399999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.45800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:30:58.904 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:02.365 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:02.368 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:02.385 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":772,"stamp":"2024-09-18T14:31:02.312122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595952,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147728,"statfs":{"total":193273528320,"available":192663273472,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1912,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":956,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003563"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":9645,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:01.433467+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:01.433467+0000","last_peered":"2024-09-18T14:31:01.433467+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:01.433467+0000","last_undegraded":"2024-09-18T14:31:01.433467+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5482,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1712},{"shards":"2,3","objects":2741}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051658,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.88}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705965,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.626},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.749}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.78400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:02.386 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:02.677 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:02.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":772,"stamp":"2024-09-18T14:31:02.312122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595952,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147728,"statfs":{"total":193273528320,"available":192663273472,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1912,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":956,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003563"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":9645,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:01.433467+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:01.433467+0000","last_peered":"2024-09-18T14:31:01.433467+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:01.433467+0000","last_undegraded":"2024-09-18T14:31:01.433467+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5482,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1712},{"shards":"2,3","objects":2741}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051658,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.88}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705965,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.626},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.749}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.78400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:02.692 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:02.692 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:02.692 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:31:02.692 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 9645, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:01.433467+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:31:01.433467+0000', 'last_peered': '2024-09-18T14:31:01.433467+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:31:01.433467+0000', 'last_undegraded': '2024-09-18T14:31:01.433467+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5482, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1712, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1712}, {'shards': '2,3', 'objects': 2741}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:02.692 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:03.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:03.058 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1473,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":610254848,"bytes_avail":192663273472,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":10075,"misplaced_total":72472,"misplaced_ratio":0.1390192074180373,"recovering_objects_per_sec":79,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":956,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:03.075 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:03.075 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:03.397 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:03.414 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":772,"stamp":"2024-09-18T14:31:02.312122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595952,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147728,"statfs":{"total":193273528320,"available":192663273472,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1912,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":956,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003563"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":9645,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:01.433467+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:01.433467+0000","last_peered":"2024-09-18T14:31:01.433467+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:01.433467+0000","last_undegraded":"2024-09-18T14:31:01.433467+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5482,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1712},{"shards":"2,3","objects":2741}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051658,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.88}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705965,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.626},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.749}]},{"osd":2,"last update":"Wed Sep 18 14:29:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.53800000000000003,"15min":0.504},"min":{"1min":0.30199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.84199999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.57599999999999996,"15min":0.54100000000000004},"min":{"1min":0.34300000000000003,"5min":0.247,"15min":0.247},"max":{"1min":0.88900000000000001,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.78400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.442}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:06.415 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:06.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:06.739 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:06.753 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":774,"stamp":"2024-09-18T14:31:06.312940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595952,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147728,"statfs":{"total":193273528320,"available":192663273472,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":9645,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:01.433467+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:01.433467+0000","last_peered":"2024-09-18T14:31:01.433467+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:01.433467+0000","last_undegraded":"2024-09-18T14:31:01.433467+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5482,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1712},{"shards":"2,3","objects":2741}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051659,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.88}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":1.008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:06.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:07.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:07.075 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":774,"stamp":"2024-09-18T14:31:06.312940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595952,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147728,"statfs":{"total":193273528320,"available":192663273472,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":9645,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:01.433467+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:01.433467+0000","last_peered":"2024-09-18T14:31:01.433467+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:01.433467+0000","last_undegraded":"2024-09-18T14:31:01.433467+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5482,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1712},{"shards":"2,3","objects":2741}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051659,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.88}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":1.008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:07.076 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:07.076 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:07.076 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:31:07.076 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 9645, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:01.433467+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:31:01.433467+0000', 'last_peered': '2024-09-18T14:31:01.433467+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:31:01.433467+0000', 'last_undegraded': '2024-09-18T14:31:01.433467+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5482, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1712, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1712}, {'shards': '2,3', 'objects': 2741}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:07.077 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:07.433 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:07.434 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1478,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":610254848,"bytes_avail":192663273472,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":10075,"misplaced_total":72472,"misplaced_ratio":0.1390192074180373,"recovering_objects_per_sec":54,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":660,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:07.449 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:07.450 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:07.761 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:07.763 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:07.774 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":774,"stamp":"2024-09-18T14:31:06.312940+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595952,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147728,"statfs":{"total":193273528320,"available":192663273472,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":660,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003187"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":9645,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:01.433467+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:01.433467+0000","last_peered":"2024-09-18T14:31:01.433467+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:01.433467+0000","last_undegraded":"2024-09-18T14:31:01.433467+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5482,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1712,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1712},{"shards":"2,3","objects":2741}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":10075,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27843,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1155072,"data_stored":905128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051659,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.78000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.88}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.74399999999999999}]}]},{"osd":0,"up_from":9,"seq":38654705966,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297976,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073864,"statfs":{"total":96636764160,"available":96331636736,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.66200000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":1.008}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270336,"data_stored":209899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:10.776 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:11.184 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:11.187 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:11.198 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":776,"stamp":"2024-09-18T14:31:10.314064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":9353,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28204,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604272,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139408,"statfs":{"total":193273528320,"available":192654753792,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1392,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003136"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":10728,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:06.444460+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:06.444460+0000","last_peered":"2024-09-18T14:31:06.444460+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:06.444460+0000","last_undegraded":"2024-09-18T14:31:06.444460+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4760,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2073},{"shards":"2,3","objects":2380}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":9353,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28204,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051660,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705967,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.56999999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.47899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705967,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.505},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.40200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.62},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.44}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:11.199 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:11.504 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:11.509 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:11.523 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":776,"stamp":"2024-09-18T14:31:10.314064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":9353,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28204,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604272,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139408,"statfs":{"total":193273528320,"available":192654753792,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1392,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003136"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":10728,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:06.444460+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:06.444460+0000","last_peered":"2024-09-18T14:31:06.444460+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:06.444460+0000","last_undegraded":"2024-09-18T14:31:06.444460+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4760,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2073},{"shards":"2,3","objects":2380}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":9353,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28204,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051660,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705967,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.56999999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.47899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705967,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.505},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.40200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.62},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.44}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:11.523 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:11.524 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:11.524 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:31:11.524 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 10728, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:06.444460+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:31:06.444460+0000', 'last_peered': '2024-09-18T14:31:06.444460+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:31:06.444460+0000', 'last_undegraded': '2024-09-18T14:31:06.444460+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4760, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2073, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2073}, {'shards': '2,3', 'objects': 2380}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:11.524 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:11.897 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:11.897 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1482,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618774528,"bytes_avail":192654753792,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":9353,"misplaced_total":72472,"misplaced_ratio":0.12905673915443205,"recovering_objects_per_sec":57,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":696,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:11.912 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:11.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:12.231 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:12.233 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:12.244 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":776,"stamp":"2024-09-18T14:31:10.314064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":9353,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28204,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604272,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139408,"statfs":{"total":193273528320,"available":192654753792,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1392,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":696,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003136"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":10728,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:06.444460+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:06.444460+0000","last_peered":"2024-09-18T14:31:06.444460+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:06.444460+0000","last_undegraded":"2024-09-18T14:31:06.444460+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4760,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2073,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2073},{"shards":"2,3","objects":2380}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":9353,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28204,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051660,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705967,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.41099999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.56999999999999995}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.47899999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705967,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.505},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.52100000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.40200000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:30:11 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61699999999999999,"15min":0.61499999999999999},"min":{"1min":0.42699999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.83999999999999997,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.62},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.622,"15min":0.61899999999999999},"min":{"1min":0.40100000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.44}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:15.246 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:15.560 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:15.562 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:15.573 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":778,"stamp":"2024-09-18T14:31:14.314988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":8567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28597,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604272,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139408,"statfs":{"total":193273528320,"available":192654753792,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1508,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002866"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":11907,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:11.443428+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:11.443428+0000","last_peered":"2024-09-18T14:31:11.443428+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:11.443428+0000","last_undegraded":"2024-09-18T14:31:11.443428+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3974,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2466,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2466},{"shards":"2,3","objects":1987}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":8567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28597,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051661,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.58299999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.77100000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:15.574 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:15.881 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:15.897 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":778,"stamp":"2024-09-18T14:31:14.314988+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":8567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28597,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604272,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139408,"statfs":{"total":193273528320,"available":192654753792,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1508,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002866"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":11907,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:11.443428+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:11.443428+0000","last_peered":"2024-09-18T14:31:11.443428+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:11.443428+0000","last_undegraded":"2024-09-18T14:31:11.443428+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3974,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2466,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2466},{"shards":"2,3","objects":1987}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":8567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28597,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051661,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.58299999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.77100000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:15.897 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:15.898 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:15.898 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:31:15.898 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 11907, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:11.443428+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:31:11.443428+0000', 'last_peered': '2024-09-18T14:31:11.443428+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:31:11.443428+0000', 'last_undegraded': '2024-09-18T14:31:11.443428+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3974, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2466, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2466}, {'shards': '2,3', 'objects': 1987}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:15.898 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:16.296 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:16.296 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1487,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618774528,"bytes_avail":192654753792,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":8567,"misplaced_total":72472,"misplaced_ratio":0.11821117121095044,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:16.309 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:16.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:16.617 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:16.620 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:16.631 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":779,"stamp":"2024-09-18T14:31:16.315428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":8567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28597,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604272,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139408,"statfs":{"total":193273528320,"available":192654753792,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1508,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":754,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002908"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":11907,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:11.443428+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:11.443428+0000","last_peered":"2024-09-18T14:31:11.443428+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:11.443428+0000","last_undegraded":"2024-09-18T14:31:11.443428+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3974,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2466,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2466},{"shards":"2,3","objects":1987}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":8567,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28597,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051661,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182478,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705968,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.58299999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:30:09 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.56299999999999994,"15min":0.54100000000000004},"min":{"1min":0.28999999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.76100000000000001,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.66900000000000004,"15min":0.63500000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.77100000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705968,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302136,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069704,"statfs":{"total":96636764160,"available":96327376896,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.40799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.51800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:19.633 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:19.964 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:19.967 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:19.978 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":780,"stamp":"2024-09-18T14:31:18.315796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":7707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29027,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2368,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002855"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":13197,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:16.449031+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:16.449031+0000","last_peered":"2024-09-18T14:31:16.449031+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:16.449031+0000","last_undegraded":"2024-09-18T14:31:16.449031+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3114,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2896},{"shards":"2,3","objects":1557}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":7707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29027,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051662,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182479,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705969,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.501},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705969,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.308},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:19.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:20.288 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:20.292 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:20.308 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":780,"stamp":"2024-09-18T14:31:18.315796+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":7707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29027,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2368,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1184,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002855"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":13197,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:16.449031+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:16.449031+0000","last_peered":"2024-09-18T14:31:16.449031+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:16.449031+0000","last_undegraded":"2024-09-18T14:31:16.449031+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3114,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2896},{"shards":"2,3","objects":1557}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":7707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29027,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051662,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182479,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705969,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.501},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705969,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.308},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:20.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:20.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:20.309 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:31:20.309 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 13197, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:16.449031+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:31:16.449031+0000', 'last_peered': '2024-09-18T14:31:16.449031+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:31:16.449031+0000', 'last_undegraded': '2024-09-18T14:31:16.449031+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3114, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2896, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2896}, {'shards': '2,3', 'objects': 1557}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:20.309 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:20.691 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:20.691 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1491,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":627294208,"bytes_avail":192646234112,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":7707,"misplaced_total":72472,"misplaced_ratio":0.10634451926261176,"recovering_objects_per_sec":68,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":823,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:20.709 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:20.709 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:21.028 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:21.041 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":781,"stamp":"2024-09-18T14:31:20.316351+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":7707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29027,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1646,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":823,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002863"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":13197,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:16.449031+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:16.449031+0000","last_peered":"2024-09-18T14:31:16.449031+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:16.449031+0000","last_undegraded":"2024-09-18T14:31:16.449031+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3114,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2896,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2896},{"shards":"2,3","objects":1557}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":7707,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29027,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1138688,"data_stored":892012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051662,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182479,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705969,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.42199999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.501},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.55700000000000005}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67300000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705969,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.46500000000000002,"15min":0.51900000000000002},"min":{"1min":0.27300000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.70899999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.308},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57499999999999996,"15min":0.63300000000000001},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.71799999999999997,"5min":1.542,"15min":1.542},"last":0.63}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.42899999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68700000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:24.043 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:24.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:24.374 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:24.388 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":783,"stamp":"2024-09-18T14:31:24.317275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":6705,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29528,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1862,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":14700,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:21.451389+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:21.451389+0000","last_peered":"2024-09-18T14:31:21.451389+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:21.451389+0000","last_undegraded":"2024-09-18T14:31:21.451389+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2112,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3397},{"shards":"2,3","objects":1056}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":6705,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29528,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1105920,"data_stored":865780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051663,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182480,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705970,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.54100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.753}]}]},{"osd":0,"up_from":9,"seq":38654705970,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.32800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.624},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":170551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:24.389 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:24.699 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:24.716 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":783,"stamp":"2024-09-18T14:31:24.317275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":6705,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29528,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1862,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":14700,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:21.451389+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:21.451389+0000","last_peered":"2024-09-18T14:31:21.451389+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:21.451389+0000","last_undegraded":"2024-09-18T14:31:21.451389+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2112,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3397},{"shards":"2,3","objects":1056}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":6705,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29528,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1105920,"data_stored":865780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051663,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182480,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705970,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.54100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.753}]}]},{"osd":0,"up_from":9,"seq":38654705970,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.32800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.624},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":170551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:24.717 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:24.717 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:24.717 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:31:24.717 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 14700, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:21.451389+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:31:21.451389+0000', 'last_peered': '2024-09-18T14:31:21.451389+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:31:21.451389+0000', 'last_undegraded': '2024-09-18T14:31:21.451389+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2112, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3397, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3397}, {'shards': '2,3', 'objects': 1056}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:24.718 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:25.106 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:25.106 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1495,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":43,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":2},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":627294208,"bytes_avail":192646234112,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":6705,"misplaced_total":72472,"misplaced_ratio":0.092518489899547418,"recovering_objects_per_sec":77,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":931,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:25.121 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:25.121 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:25.446 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:25.448 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:25.463 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":783,"stamp":"2024-09-18T14:31:24.317275+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":6705,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29528,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612592,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188131088,"statfs":{"total":193273528320,"available":192646234112,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1862,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":931,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002670"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":14700,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:21.451389+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:21.451389+0000","last_peered":"2024-09-18T14:31:21.451389+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:21.451389+0000","last_undegraded":"2024-09-18T14:31:21.451389+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2112,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3397},{"shards":"2,3","objects":1056}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":6705,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29528,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1105920,"data_stored":865780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051663,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182480,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705970,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:30:21 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.53200000000000003},"min":{"1min":0.29299999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.70899999999999996,"5min":0.97299999999999998,"15min":6.758},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.56599999999999995,"15min":0.60999999999999999},"min":{"1min":0.432,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.84599999999999997,"5min":0.89200000000000002,"15min":6.7859999999999996},"last":0.67300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.497},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.54100000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.753}]}]},{"osd":0,"up_from":9,"seq":38654705970,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306296,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065544,"statfs":{"total":96636764160,"available":96323117056,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.32800000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.58299999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.624},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.437}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":170551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:28.465 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:28.805 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:28.809 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:28.819 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":785,"stamp":"2024-09-18T14:31:28.318520+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":5391,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30185,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-3176,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1588,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003091"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":16671,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:26.452819+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:26.452819+0000","last_peered":"2024-09-18T14:31:26.452819+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:26.452819+0000","last_undegraded":"2024-09-18T14:31:26.452819+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":798,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4054},{"shards":"2,3","objects":399}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":5391,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30185,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":852664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051664,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182481,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.441}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.373}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.432}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:28.820 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:29.148 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:29.151 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:29.161 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":786,"stamp":"2024-09-18T14:31:29.098636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":5391,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30185,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.782839"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":16671,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:26.452819+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:26.452819+0000","last_peered":"2024-09-18T14:31:26.452819+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:26.452819+0000","last_undegraded":"2024-09-18T14:31:26.452819+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":798,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4054},{"shards":"2,3","objects":399}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":5391,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30185,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":852664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051664,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182481,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.441}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.373}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.432}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:29.162 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:29.162 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 4642, 'reported_epoch': 42, 'state': 'active+recovery_wait+undersized+degraded+remapped', 'last_fresh': '2024-09-18T14:30:33.709656+0000', 'last_change': '2024-09-18T14:24:48.474711+0000', 'last_active': '2024-09-18T14:30:33.709656+0000', 'last_peered': '2024-09-18T14:30:33.709656+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:30:33.709656+0000', 'last_undegraded': '2024-09-18T14:24:48.457522+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 4593, 'num_objects_misplaced': 4593, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2,3', 'objects': 4593}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:29.162 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:31:29.162 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 16671, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:26.452819+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:31:26.452819+0000', 'last_peered': '2024-09-18T14:31:26.452819+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:31:26.452819+0000', 'last_undegraded': '2024-09-18T14:31:26.452819+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 798, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4054, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 4054}, {'shards': '2,3', 'objects': 399}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:29.162 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:29.550 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:29.550 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 4593/72472 objects degraded (6.338%), 1 pg degraded, 2 pgs undersized","count":3},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1500,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":44,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":1},{"state_name":"active+recovery_wait+undersized+degraded+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635813888,"bytes_avail":192637714432,"bytes_total":193273528320,"degraded_objects":4593,"degraded_total":72472,"degraded_ratio":0.063376200463627338,"misplaced_objects":5391,"misplaced_total":72472,"misplaced_ratio":0.074387349597085775,"recovering_objects_per_sec":132,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1588,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:29.562 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:29.562 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:29.846 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:29.849 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:29.863 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":786,"stamp":"2024-09-18T14:31:29.098636+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":5391,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30185,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620912,"kb_used_data":1248,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122768,"statfs":{"total":193273528320,"available":192637714432,"internally_reserved":0,"allocated":1277952,"data_stored":711115,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2316,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1158,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.782839"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4642,"reported_epoch":42,"state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2024-09-18T14:30:33.709656+0000","last_change":"2024-09-18T14:24:48.474711+0000","last_active":"2024-09-18T14:30:33.709656+0000","last_peered":"2024-09-18T14:30:33.709656+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:30:33.709656+0000","last_undegraded":"2024-09-18T14:24:48.457522+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":4593,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2,3","objects":4593}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":16671,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:26.452819+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:26.452819+0000","last_peered":"2024-09-18T14:31:26.452819+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:26.452819+0000","last_undegraded":"2024-09-18T14:31:26.452819+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":798,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4054},{"shards":"2,3","objects":399}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4532,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702830+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:30:33.702830+0000","last_peered":"2024-09-18T14:30:33.702830+0000","last_clean":"2024-09-18T14:30:33.702830+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:30:33.702830+0000","last_undegraded":"2024-09-18T14:30:33.702830+0000","last_fullsized":"2024-09-18T14:30:33.702830+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4565,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.702852+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:30:33.702852+0000","last_peered":"2024-09-18T14:30:33.702852+0000","last_clean":"2024-09-18T14:30:33.702852+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:30:33.702852+0000","last_undegraded":"2024-09-18T14:30:33.702852+0000","last_fullsized":"2024-09-18T14:30:33.702852+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4593,"num_objects_misplaced":5391,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30185,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1089536,"data_stored":852664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051664,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182481,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.441}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.373}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.67500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.432}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:32.865 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:33.179 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:33.183 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:33.195 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-09-18T14:31:32.319462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9688,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30333,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620964,"kb_used_data":1300,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122716,"statfs":{"total":193273528320,"available":192637661184,"internally_reserved":0,"allocated":1331200,"data_stored":743806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":2983,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002187"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":5089,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:31.436217+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:31.436217+0000","last_peered":"2024-09-18T14:31:31.436217+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:31.436217+0000","last_undegraded":"2024-09-18T14:31:31.436217+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":148},{"shards":"2,3","objects":4445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":16671,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:26.452819+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:26.452819+0000","last_peered":"2024-09-18T14:31:26.452819+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:26.452819+0000","last_undegraded":"2024-09-18T14:31:26.452819+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":798,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4054},{"shards":"2,3","objects":399}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9688,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30333,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":878920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051664,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182482,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705972,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.376},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.432}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:33.196 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:33.530 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:33.533 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:33.545 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-09-18T14:31:32.319462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9688,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30333,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620964,"kb_used_data":1300,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122716,"statfs":{"total":193273528320,"available":192637661184,"internally_reserved":0,"allocated":1331200,"data_stored":743806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":2983,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002187"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":5089,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:31.436217+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:31.436217+0000","last_peered":"2024-09-18T14:31:31.436217+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:31.436217+0000","last_undegraded":"2024-09-18T14:31:31.436217+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":148},{"shards":"2,3","objects":4445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":16671,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:26.452819+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:26.452819+0000","last_peered":"2024-09-18T14:31:26.452819+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:26.452819+0000","last_undegraded":"2024-09-18T14:31:26.452819+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":798,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4054},{"shards":"2,3","objects":399}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9688,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30333,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":878920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051664,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182482,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705972,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.376},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.432}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:33.545 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:33.545 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 5089, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:31.436217+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:31:31.436217+0000', 'last_peered': '2024-09-18T14:31:31.436217+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:31:31.436217+0000', 'last_undegraded': '2024-09-18T14:31:31.436217+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8890, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 148, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 148}, {'shards': '2,3', 'objects': 4445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:33.546 INFO:tasks.osd_recovery.ceph_manager:PG 1.5 is not active+clean 2024-09-18T14:31:33.546 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.5', 'version': "17'4453", 'reported_seq': 16671, 'reported_epoch': 43, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:26.452819+0000', 'last_change': '2024-09-18T14:30:32.778475+0000', 'last_active': '2024-09-18T14:31:26.452819+0000', 'last_peered': '2024-09-18T14:31:26.452819+0000', 'last_clean': '2024-09-18T14:24:00.766786+0000', 'last_became_active': '2024-09-18T14:24:48.465182+0000', 'last_became_peered': '2024-09-18T14:24:48.465182+0000', 'last_unstale': '2024-09-18T14:31:26.452819+0000', 'last_undegraded': '2024-09-18T14:31:26.452819+0000', 'last_fullsized': '2024-09-18T14:24:48.315910+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 12, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4453, 'log_dups_size': 0, 'ondisk_log_size': 4453, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0.000740488, 'stat_sum': {'num_bytes': 30706, 'num_objects': 4453, 'num_object_clones': 0, 'num_object_copies': 8906, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 798, 'num_objects_unfound': 0, 'num_objects_dirty': 4453, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4453, 'num_write_kb': 4477, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4054, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 4054}, {'shards': '2,3', 'objects': 399}], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:33.546 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:33.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:33.896 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 2 pgs undersized","count":2},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1504,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":6},{"state_name":"active+recovering+undersized+remapped","count":2}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635867136,"bytes_avail":192637661184,"bytes_total":193273528320,"misplaced_objects":9688,"misplaced_total":72472,"misplaced_ratio":0.13367921404128491,"recovering_objects_per_sec":100,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":805,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:33.910 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:33.910 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:34.215 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:34.218 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:34.230 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":789,"stamp":"2024-09-18T14:31:32.319462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9688,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30333,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":620964,"kb_used_data":1300,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122716,"statfs":{"total":193273528320,"available":192637661184,"internally_reserved":0,"allocated":1331200,"data_stored":743806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":2983,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":805,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002187"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":5089,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:31.436217+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:31.436217+0000","last_peered":"2024-09-18T14:31:31.436217+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:31.436217+0000","last_undegraded":"2024-09-18T14:31:31.436217+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":148},{"shards":"2,3","objects":4445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4606,"reported_epoch":43,"state":"active+clean","last_fresh":"2024-09-18T14:30:34.719268+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:30:34.719268+0000","last_peered":"2024-09-18T14:30:34.719268+0000","last_clean":"2024-09-18T14:30:34.719268+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:30:34.719268+0000","last_undegraded":"2024-09-18T14:30:34.719268+0000","last_fullsized":"2024-09-18T14:30:34.719268+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":16671,"reported_epoch":43,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:26.452819+0000","last_change":"2024-09-18T14:30:32.778475+0000","last_active":"2024-09-18T14:31:26.452819+0000","last_peered":"2024-09-18T14:31:26.452819+0000","last_clean":"2024-09-18T14:24:00.766786+0000","last_became_active":"2024-09-18T14:24:48.465182+0000","last_became_peered":"2024-09-18T14:24:48.465182+0000","last_unstale":"2024-09-18T14:31:26.452819+0000","last_undegraded":"2024-09-18T14:31:26.452819+0000","last_fullsized":"2024-09-18T14:24:48.315910+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T19:04:30.554551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":798,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4054,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4054},{"shards":"2,3","objects":399}],"blocked_by":[],"up_primary":0,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4569,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705293+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:30:33.705293+0000","last_peered":"2024-09-18T14:30:33.705293+0000","last_clean":"2024-09-18T14:30:33.705293+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:30:33.705293+0000","last_undegraded":"2024-09-18T14:30:33.705293+0000","last_fullsized":"2024-09-18T14:30:33.705293+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4682,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705183+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:30:33.705183+0000","last_peered":"2024-09-18T14:30:33.705183+0000","last_clean":"2024-09-18T14:30:33.705183+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:30:33.705183+0000","last_undegraded":"2024-09-18T14:30:33.705183+0000","last_fullsized":"2024-09-18T14:30:33.705183+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4552,"reported_epoch":42,"state":"active+clean","last_fresh":"2024-09-18T14:30:33.705081+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:30:33.705081+0000","last_peered":"2024-09-18T14:30:33.705081+0000","last_clean":"2024-09-18T14:30:33.705081+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:30:33.705081+0000","last_undegraded":"2024-09-18T14:30:33.705081+0000","last_fullsized":"2024-09-18T14:30:33.705081+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9688,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30333,"num_bytes_recovered":223010,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1126400,"data_stored":878920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":14,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051664,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182482,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705972,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.376},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705971,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310456,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061384,"statfs":{"total":96636764160,"available":96318857216,"internally_reserved":0,"allocated":638976,"data_stored":355428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.58799999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.432}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.54000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":222991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:37.231 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:37.716 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:37.720 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:37.732 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":791,"stamp":"2024-09-18T14:31:36.320411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30732,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":3499,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":547,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001891"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":5089,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:31.436217+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:31.436217+0000","last_peered":"2024-09-18T14:31:31.436217+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:31.436217+0000","last_undegraded":"2024-09-18T14:31:31.436217+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":148},{"shards":"2,3","objects":4445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30732,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163264,"data_stored":905176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051665,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182482,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705972,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.376},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705972,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.495}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.32000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.53500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:37.733 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:38.042 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:38.045 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:38.055 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":791,"stamp":"2024-09-18T14:31:36.320411+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30732,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":3499,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":547,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001891"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":5089,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:31.436217+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:31.436217+0000","last_peered":"2024-09-18T14:31:31.436217+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:31.436217+0000","last_undegraded":"2024-09-18T14:31:31.436217+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":148,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":148},{"shards":"2,3","objects":4445}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8890,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30732,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163264,"data_stored":905176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051665,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182482,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705972,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48399999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.376},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.79500000000000004}]}]},{"osd":0,"up_from":9,"seq":38654705972,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.495}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.32000000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.53500000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:38.056 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:38.056 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 5089, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:31.436217+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:31:31.436217+0000', 'last_peered': '2024-09-18T14:31:31.436217+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:31:31.436217+0000', 'last_undegraded': '2024-09-18T14:31:31.436217+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 8890, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 148, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 148}, {'shards': '2,3', 'objects': 4445}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:38.056 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:38.418 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:38.419 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1509,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635920384,"bytes_avail":192637607936,"bytes_total":193273528320,"misplaced_objects":8890,"misplaced_total":72472,"misplaced_ratio":0.12266806490782647,"recovering_objects_per_sec":68,"recovering_bytes_per_sec":3281,"recovering_keys_per_sec":0,"num_objects_recovered":547,"num_bytes_recovered":26256,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:38.432 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:38.433 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:38.904 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:38.920 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":792,"stamp":"2024-09-18T14:31:38.320784+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8370,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30992,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":2979,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":807,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.222147"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":5869,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:36.456442+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:36.456442+0000","last_peered":"2024-09-18T14:31:36.456442+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:36.456442+0000","last_undegraded":"2024-09-18T14:31:36.456442+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8370,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":408},{"shards":"2,3","objects":4185}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8370,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30992,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163264,"data_stored":905176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182483,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705973,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.628},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.73299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705973,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.503},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.77400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.45800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:41.922 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:42.222 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:42.226 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:42.236 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":793,"stamp":"2024-09-18T14:31:40.321175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8370,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30992,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":2979,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":807,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.220373"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":5869,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:36.456442+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:36.456442+0000","last_peered":"2024-09-18T14:31:36.456442+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:36.456442+0000","last_undegraded":"2024-09-18T14:31:36.456442+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8370,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":408,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":408},{"shards":"2,3","objects":4185}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8370,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30992,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163264,"data_stored":905176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182483,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705973,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.628},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.67800000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.73299999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705973,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.503},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.77400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.45800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:42.237 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:42.542 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:42.545 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:42.557 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":794,"stamp":"2024-09-18T14:31:42.321838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":608536,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135144,"statfs":{"total":193273528320,"available":192650387456,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":2433,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002884"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":6688,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:41.459272+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:41.459272+0000","last_peered":"2024-09-18T14:31:41.459272+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:41.459272+0000","last_undegraded":"2024-09-18T14:31:41.459272+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":681},{"shards":"2,3","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163264,"data_stored":905176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182484,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.63200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.379},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705973,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.503},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.77400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.45800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:42.558 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:42.558 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 6688, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:41.459272+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:31:41.459272+0000', 'last_peered': '2024-09-18T14:31:41.459272+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:31:41.459272+0000', 'last_undegraded': '2024-09-18T14:31:41.459272+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7824, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 681, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 681}, {'shards': '2,3', 'objects': 3912}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:42.559 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:42.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:42.932 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1513,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":623140864,"bytes_avail":192650387456,"bytes_total":193273528320,"misplaced_objects":7824,"misplaced_total":72472,"misplaced_ratio":0.10795893586488575,"recovering_objects_per_sec":89,"recovering_bytes_per_sec":2187,"recovering_keys_per_sec":0,"num_objects_recovered":1080,"num_bytes_recovered":26256,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:42.952 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:42.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:43.249 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:43.252 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:43.262 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":794,"stamp":"2024-09-18T14:31:42.321838+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":608536,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135144,"statfs":{"total":193273528320,"available":192650387456,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-4593,"num_objects_misplaced":2433,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1080,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002884"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":6688,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:41.459272+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:41.459272+0000","last_peered":"2024-09-18T14:31:41.459272+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:41.459272+0000","last_undegraded":"2024-09-18T14:31:41.459272+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":681},{"shards":"2,3","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1163264,"data_stored":905176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051666,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182484,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.63200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.379},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705973,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.503},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.77400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.45800000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.61699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:46.264 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:46.578 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:46.581 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:46.593 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":796,"stamp":"2024-09-18T14:31:46.322897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1066,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002887"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":6688,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:41.459272+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:41.459272+0000","last_peered":"2024-09-18T14:31:41.459272+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:41.459272+0000","last_undegraded":"2024-09-18T14:31:41.459272+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":681},{"shards":"2,3","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1130496,"data_stored":878944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051667,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182484,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.63200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.379},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.625}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":131203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:46.594 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:46.989 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:47.004 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":796,"stamp":"2024-09-18T14:31:46.322897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1066,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002887"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":6688,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:41.459272+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:41.459272+0000","last_peered":"2024-09-18T14:31:41.459272+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:41.459272+0000","last_undegraded":"2024-09-18T14:31:41.459272+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":681},{"shards":"2,3","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1130496,"data_stored":878944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051667,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182484,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.63200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.379},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.625}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":131203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:47.005 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:47.005 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 6688, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:41.459272+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:31:41.459272+0000', 'last_peered': '2024-09-18T14:31:41.459272+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:31:41.459272+0000', 'last_undegraded': '2024-09-18T14:31:41.459272+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7824, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 681, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 681}, {'shards': '2,3', 'objects': 3912}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:47.005 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:47.400 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:47.400 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1518,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":610361344,"bytes_avail":192663166976,"bytes_total":193273528320,"misplaced_objects":7824,"misplaced_total":72472,"misplaced_ratio":0.10795893586488575,"recovering_objects_per_sec":44,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":533,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:47.416 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:47.416 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:47.772 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:47.775 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:47.790 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":796,"stamp":"2024-09-18T14:31:46.322897+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1066,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":533,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002887"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":6688,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:41.459272+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:41.459272+0000","last_peered":"2024-09-18T14:31:41.459272+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:41.459272+0000","last_undegraded":"2024-09-18T14:31:41.459272+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":681,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":681},{"shards":"2,3","objects":3912}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7824,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31265,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1130496,"data_stored":878944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051667,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182484,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.63200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.379},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.72199999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705974,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.625}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.44700000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.66400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":131203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:50.792 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:51.290 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:51.293 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:51.304 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2024-09-18T14:31:50.324118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7244,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31555,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1126,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003334"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":7558,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:46.459742+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:46.459742+0000","last_peered":"2024-09-18T14:31:46.459742+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:46.459742+0000","last_undegraded":"2024-09-18T14:31:46.459742+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7244,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":971},{"shards":"2,3","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7244,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31555,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1130496,"data_stored":878944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051668,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182485,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705975,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.66500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.379},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.628}]}]},{"osd":0,"up_from":9,"seq":38654705975,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.79500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":131203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:51.305 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:51.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:51.927 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:51.938 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":798,"stamp":"2024-09-18T14:31:50.324118+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7244,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31555,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1126,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003334"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":7558,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:46.459742+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:46.459742+0000","last_peered":"2024-09-18T14:31:46.459742+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:46.459742+0000","last_undegraded":"2024-09-18T14:31:46.459742+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7244,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":971,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":971},{"shards":"2,3","objects":3622}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7244,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31555,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1130496,"data_stored":878944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051668,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182485,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705975,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.66500000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.379},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.53800000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.628}]}]},{"osd":0,"up_from":9,"seq":38654705975,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.79500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":131203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:51.938 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:51.938 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 7558, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:46.459742+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:31:46.459742+0000', 'last_peered': '2024-09-18T14:31:46.459742+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:31:46.459742+0000', 'last_undegraded': '2024-09-18T14:31:46.459742+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 7244, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 971, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 971}, {'shards': '2,3', 'objects': 3622}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:51.939 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:52.278 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:52.279 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1523,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":610361344,"bytes_avail":192663166976,"bytes_total":193273528320,"misplaced_objects":7244,"misplaced_total":72472,"misplaced_ratio":0.099955845016006184,"recovering_objects_per_sec":46,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":563,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:52.292 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:52.292 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:52.598 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:52.603 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:52.617 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":799,"stamp":"2024-09-18T14:31:52.324625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31859,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1734,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":867,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003450"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":8470,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:51.462045+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:51.462045+0000","last_peered":"2024-09-18T14:31:51.462045+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:51.462045+0000","last_undegraded":"2024-09-18T14:31:51.462045+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1275},{"shards":"2,3","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31859,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1130496,"data_stored":878944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051668,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182486,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705976,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.496},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705975,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.60799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.36099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.79500000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":131203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:55.619 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:55.917 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:55.920 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:55.930 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2024-09-18T14:31:54.324992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31859,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1188,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":594,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003153"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":8470,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:51.462045+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:51.462045+0000","last_peered":"2024-09-18T14:31:51.462045+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:51.462045+0000","last_undegraded":"2024-09-18T14:31:51.462045+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1275},{"shards":"2,3","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31859,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":865828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051669,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182486,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705976,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.496},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705976,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.52600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":118087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:55.931 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:56.242 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:56.245 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:56.260 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":800,"stamp":"2024-09-18T14:31:54.324992+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31859,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1188,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":594,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003153"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":8470,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:51.462045+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:51.462045+0000","last_peered":"2024-09-18T14:31:51.462045+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:51.462045+0000","last_undegraded":"2024-09-18T14:31:51.462045+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1275},{"shards":"2,3","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31859,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":865828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051669,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182486,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705976,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.496},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705976,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.52600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":118087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:56.261 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:31:56.261 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 8470, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:51.462045+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:31:51.462045+0000', 'last_peered': '2024-09-18T14:31:51.462045+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:31:51.462045+0000', 'last_undegraded': '2024-09-18T14:31:51.462045+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6636, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1275, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1275}, {'shards': '2,3', 'objects': 3318}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:31:56.261 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:31:56.658 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:56.659 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1527,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":610361344,"bytes_avail":192663166976,"bytes_total":193273528320,"misplaced_objects":6636,"misplaced_total":72472,"misplaced_ratio":0.091566398057180698,"recovering_objects_per_sec":49,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":594,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:31:56.676 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:31:56.677 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:31:56.965 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:31:56.968 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:31:56.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":801,"stamp":"2024-09-18T14:31:56.325495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31859,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596056,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147624,"statfs":{"total":193273528320,"available":192663166976,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1188,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":594,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002996"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":8470,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:51.462045+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:51.462045+0000","last_peered":"2024-09-18T14:31:51.462045+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:51.462045+0000","last_undegraded":"2024-09-18T14:31:51.462045+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1275,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1275},{"shards":"2,3","objects":3318}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6636,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31859,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":865828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051669,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182486,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705976,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.624}]},{"osd":2,"last update":"Wed Sep 18 14:30:51 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.51400000000000001,"15min":0.48499999999999999},"min":{"1min":0.32900000000000001,"5min":0.32800000000000001,"15min":0.32000000000000001},"max":{"1min":0.64900000000000002,"5min":1.357,"15min":1.357},"last":0.496},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.56499999999999995,"15min":2.266},"min":{"1min":0.39200000000000002,"5min":0.31,"15min":0.26600000000000001},"max":{"1min":0.78600000000000003,"5min":0.93100000000000005,"15min":46.807000000000002},"last":0.43099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.52500000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705976,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298028,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073812,"statfs":{"total":96636764160,"available":96331583488,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.47899999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.52600000000000002}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.71999999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":118087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:31:59.986 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:00.294 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:00.299 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:00.311 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":802,"stamp":"2024-09-18T14:31:58.325879+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32159,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1788,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":894,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002982"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":9370,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:56.465836+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:56.465836+0000","last_peered":"2024-09-18T14:31:56.465836+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:56.465836+0000","last_undegraded":"2024-09-18T14:31:56.465836+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1575},{"shards":"2,3","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32159,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":865828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051670,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182487,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705977,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.38200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57599999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705977,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.92100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.54300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.78300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":118087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:00.312 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:00.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:00.611 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:00.622 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2024-09-18T14:32:00.326345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32159,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":9370,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:56.465836+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:56.465836+0000","last_peered":"2024-09-18T14:31:56.465836+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:56.465836+0000","last_undegraded":"2024-09-18T14:31:56.465836+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1575},{"shards":"2,3","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32159,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":865828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051670,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182487,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705977,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.38200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57599999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705977,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.92100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.54300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.78300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":118087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:00.623 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:32:00.623 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 9370, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:31:56.465836+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:31:56.465836+0000', 'last_peered': '2024-09-18T14:31:56.465836+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:31:56.465836+0000', 'last_undegraded': '2024-09-18T14:31:56.465836+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 6036, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1575, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1575}, {'shards': '2,3', 'objects': 3018}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:32:00.623 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:32:00.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:00.976 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1531,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618881024,"bytes_avail":192654647296,"bytes_total":193273528320,"misplaced_objects":6036,"misplaced_total":72472,"misplaced_ratio":0.083287338558339752,"recovering_objects_per_sec":50,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":604,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:32:00.991 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:32:00.991 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:01.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:01.310 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:01.321 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":803,"stamp":"2024-09-18T14:32:00.326345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32159,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1208,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":604,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002813"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":9370,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:31:56.465836+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:31:56.465836+0000","last_peered":"2024-09-18T14:31:56.465836+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:31:56.465836+0000","last_undegraded":"2024-09-18T14:31:56.465836+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1575},{"shards":"2,3","objects":3018}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6036,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32159,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1114112,"data_stored":865828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051670,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182487,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705977,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.38200000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.54600000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.57599999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705977,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.92100000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:30:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.54700000000000004,"15min":0.50700000000000001},"min":{"1min":0.34999999999999998,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.82399999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.36399999999999999,"5min":0.27900000000000003,"15min":0.247},"max":{"1min":0.85599999999999998,"5min":3.1920000000000002,"15min":3.1920000000000002},"last":0.54300000000000004}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.78300000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":118087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:04.324 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:04.657 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:04.660 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:04.670 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":805,"stamp":"2024-09-18T14:32:04.327374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32498,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1278,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":10387,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:01.469924+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:01.469924+0000","last_peered":"2024-09-18T14:32:01.469924+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:01.469924+0000","last_undegraded":"2024-09-18T14:32:01.469924+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1914,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1914},{"shards":"2,3","objects":2679}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32498,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":852712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182488,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.35999999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.62}]}]},{"osd":0,"up_from":9,"seq":38654705978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.57099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.44400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":104971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:04.671 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:04.954 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:04.957 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:04.970 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":805,"stamp":"2024-09-18T14:32:04.327374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32498,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1278,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":10387,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:01.469924+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:01.469924+0000","last_peered":"2024-09-18T14:32:01.469924+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:01.469924+0000","last_undegraded":"2024-09-18T14:32:01.469924+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1914,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1914},{"shards":"2,3","objects":2679}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32498,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":852712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182488,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.35999999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.62}]}]},{"osd":0,"up_from":9,"seq":38654705978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.57099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.44400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":104971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:04.971 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:32:04.971 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 10387, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:32:01.469924+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:32:01.469924+0000', 'last_peered': '2024-09-18T14:32:01.469924+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:32:01.469924+0000', 'last_undegraded': '2024-09-18T14:32:01.469924+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 5358, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 1914, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 1914}, {'shards': '2,3', 'objects': 2679}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:32:04.972 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:32:05.345 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:05.345 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1536,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618881024,"bytes_avail":192654647296,"bytes_total":193273528320,"misplaced_objects":5358,"misplaced_total":72472,"misplaced_ratio":0.073932001324649527,"recovering_objects_per_sec":53,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:32:05.362 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:32:05.362 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:05.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:05.695 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:05.706 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":805,"stamp":"2024-09-18T14:32:04.327374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32498,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1278,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":639,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002748"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":10387,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:01.469924+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:01.469924+0000","last_peered":"2024-09-18T14:32:01.469924+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:01.469924+0000","last_undegraded":"2024-09-18T14:32:01.469924+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1914,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":1914},{"shards":"2,3","objects":2679}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5358,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32498,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1097728,"data_stored":852712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051671,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182488,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.56299999999999994}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.35999999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.62}]}]},{"osd":0,"up_from":9,"seq":38654705978,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.57099999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.84399999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.44400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":139264,"data_stored":104971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:08.707 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:09.033 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:09.035 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:09.047 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2024-09-18T14:32:08.328142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32856,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1994,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002646"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":11462,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:06.472770+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:06.472770+0000","last_peered":"2024-09-18T14:32:06.472770+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:06.472770+0000","last_undegraded":"2024-09-18T14:32:06.472770+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2272},{"shards":"2,3","objects":2321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32856,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1081344,"data_stored":839596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182489,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705979,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81599999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705979,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.379},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.59999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":91855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:09.048 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:09.365 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:09.368 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:09.378 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2024-09-18T14:32:08.328142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32856,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1994,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002646"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":11462,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:06.472770+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:06.472770+0000","last_peered":"2024-09-18T14:32:06.472770+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:06.472770+0000","last_undegraded":"2024-09-18T14:32:06.472770+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2272},{"shards":"2,3","objects":2321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32856,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1081344,"data_stored":839596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182489,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705979,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81599999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705979,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.379},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.59999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":91855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:09.379 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:32:09.379 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 11462, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:32:06.472770+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:32:06.472770+0000', 'last_peered': '2024-09-18T14:32:06.472770+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:32:06.472770+0000', 'last_undegraded': '2024-09-18T14:32:06.472770+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 4642, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2272, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2272}, {'shards': '2,3', 'objects': 2321}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:32:09.379 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:32:09.740 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:09.740 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1540,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":618881024,"bytes_avail":192654647296,"bytes_total":193273528320,"misplaced_objects":4642,"misplaced_total":72472,"misplaced_ratio":0.064052323656032681,"recovering_objects_per_sec":83,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":997,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:32:09.754 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:32:09.754 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:10.070 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:10.073 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:10.083 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":807,"stamp":"2024-09-18T14:32:08.328142+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32856,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604376,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":603004,"kb_avail":188139304,"statfs":{"total":193273528320,"available":192654647296,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":617476784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1994,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":997,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002646"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":11462,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:06.472770+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:06.472770+0000","last_peered":"2024-09-18T14:32:06.472770+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:06.472770+0000","last_undegraded":"2024-09-18T14:32:06.472770+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2272,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2272},{"shards":"2,3","objects":2321}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4642,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32856,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1081344,"data_stored":839596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182489,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705979,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.54300000000000004}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.57499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.748},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.81599999999999995}]}]},{"osd":0,"up_from":9,"seq":38654705979,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.379},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.59999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":91855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:13.085 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:13.409 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:13.423 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":809,"stamp":"2024-09-18T14:32:12.329139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":608536,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135144,"statfs":{"total":193273528320,"available":192650387456,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2174,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002793"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":12631,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:11.475405+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:11.475405+0000","last_peered":"2024-09-18T14:32:11.475405+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:11.475405+0000","last_undegraded":"2024-09-18T14:32:11.475405+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2662},{"shards":"2,3","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1081344,"data_stored":839596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182490,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53100000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705979,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.379},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.59999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":91855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:13.424 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:13.750 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:13.754 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:13.764 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":809,"stamp":"2024-09-18T14:32:12.329139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":608536,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":607164,"kb_avail":188135144,"statfs":{"total":193273528320,"available":192650387456,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":621736624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2174,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002793"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":12631,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:11.475405+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:11.475405+0000","last_peered":"2024-09-18T14:32:11.475405+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:11.475405+0000","last_undegraded":"2024-09-18T14:32:11.475405+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2662},{"shards":"2,3","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1081344,"data_stored":839596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051672,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182490,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53100000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705979,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302188,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":94069652,"statfs":{"total":96636764160,"available":96327323648,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.379},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.59999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.47499999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.55600000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":91855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:13.766 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:32:13.766 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 12631, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:32:11.475405+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:32:11.475405+0000', 'last_peered': '2024-09-18T14:32:11.475405+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:32:11.475405+0000', 'last_undegraded': '2024-09-18T14:32:11.475405+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3862, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2662, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2662}, {'shards': '2,3', 'objects': 1931}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:32:13.767 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:32:14.132 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:14.133 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1545,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":623140864,"bytes_avail":192650387456,"bytes_total":193273528320,"misplaced_objects":3862,"misplaced_total":72472,"misplaced_ratio":0.053289546307539466,"recovering_objects_per_sec":90,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1087,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:32:14.151 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:32:14.151 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:14.471 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:14.474 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:14.489 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":810,"stamp":"2024-09-18T14:32:14.329693+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612696,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188130984,"statfs":{"total":193273528320,"available":192646127616,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1496,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":748,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002959"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":12631,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:11.475405+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:11.475405+0000","last_peered":"2024-09-18T14:32:11.475405+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:11.475405+0000","last_undegraded":"2024-09-18T14:32:11.475405+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2662},{"shards":"2,3","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":826480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182490,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53100000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":78739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:17.490 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:17.811 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:17.814 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:17.825 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":811,"stamp":"2024-09-18T14:32:16.330240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612696,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188130984,"statfs":{"total":193273528320,"available":192646127616,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1496,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":748,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002866"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":12631,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:11.475405+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:11.475405+0000","last_peered":"2024-09-18T14:32:11.475405+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:11.475405+0000","last_undegraded":"2024-09-18T14:32:11.475405+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2662},{"shards":"2,3","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":826480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182490,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53100000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":78739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:17.825 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:18.128 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:18.131 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:18.141 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":811,"stamp":"2024-09-18T14:32:16.330240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612696,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188130984,"statfs":{"total":193273528320,"available":192646127616,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1496,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":748,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002866"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":12631,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:11.475405+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:11.475405+0000","last_peered":"2024-09-18T14:32:11.475405+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:11.475405+0000","last_undegraded":"2024-09-18T14:32:11.475405+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2662,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":2662},{"shards":"2,3","objects":1931}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3862,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33246,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1064960,"data_stored":826480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051673,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182490,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48699999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.57599999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:31:13 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56399999999999995,"15min":0.54200000000000004},"min":{"1min":0.38500000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.81399999999999995,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.67700000000000005,"15min":0.63700000000000001},"min":{"1min":0.47899999999999998,"5min":0.377,"15min":0.377},"max":{"1min":0.90500000000000003,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.53100000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705980,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.59599999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:31:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.61399999999999999},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.94099999999999995,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.61199999999999999,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":78739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:18.142 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:32:18.142 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 12631, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:32:11.475405+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:32:11.475405+0000', 'last_peered': '2024-09-18T14:32:11.475405+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:32:11.475405+0000', 'last_undegraded': '2024-09-18T14:32:11.475405+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 3862, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 2662, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 2662}, {'shards': '2,3', 'objects': 1931}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:32:18.142 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:32:18.552 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:18.553 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1549,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":627400704,"bytes_avail":192646127616,"bytes_total":193273528320,"misplaced_objects":3862,"misplaced_total":72472,"misplaced_ratio":0.053289546307539466,"recovering_objects_per_sec":62,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":748,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:32:18.568 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:32:18.568 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:18.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:18.892 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:18.907 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":812,"stamp":"2024-09-18T14:32:18.330618+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3038,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33658,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612696,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188130984,"statfs":{"total":193273528320,"available":192646127616,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2320,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1160,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002850"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":13867,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:16.480450+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:16.480450+0000","last_peered":"2024-09-18T14:32:16.480450+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:16.480450+0000","last_undegraded":"2024-09-18T14:32:16.480450+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3038,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3074,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3074},{"shards":"2,3","objects":1519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3038,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33658,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182491,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705981,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.379},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.496},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705981,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.44500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:21.908 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:22.234 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:22.236 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:22.247 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":813,"stamp":"2024-09-18T14:32:20.330973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3038,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33658,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612696,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188130984,"statfs":{"total":193273528320,"available":192646127616,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1604,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":802,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002830"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":13867,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:16.480450+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:16.480450+0000","last_peered":"2024-09-18T14:32:16.480450+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:16.480450+0000","last_undegraded":"2024-09-18T14:32:16.480450+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3038,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3074,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3074},{"shards":"2,3","objects":1519}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3038,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33658,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182491,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705981,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.379},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.52400000000000002}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.44500000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.496},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.64300000000000002}]}]},{"osd":0,"up_from":9,"seq":38654705981,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.44500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:22.248 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:22.572 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:22.575 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:22.589 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":814,"stamp":"2024-09-18T14:32:22.331366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616856,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188126824,"statfs":{"total":193273528320,"available":192641867776,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2608,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002806"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":15373,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:21.483784+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:21.483784+0000","last_peered":"2024-09-18T14:32:21.483784+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:21.483784+0000","last_undegraded":"2024-09-18T14:32:21.483784+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3576},{"shards":"2,3","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705982,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.71699999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705981,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.44500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:22.590 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:32:22.590 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 15373, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:32:21.483784+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:32:21.483784+0000', 'last_peered': '2024-09-18T14:32:21.483784+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:32:21.483784+0000', 'last_undegraded': '2024-09-18T14:32:21.483784+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2034, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3576, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3576}, {'shards': '2,3', 'objects': 1017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:32:22.591 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:32:22.964 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:22.964 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1553,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":631660544,"bytes_avail":192641867776,"bytes_total":193273528320,"misplaced_objects":2034,"misplaced_total":72472,"misplaced_ratio":0.028066011701070755,"recovering_objects_per_sec":108,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:32:22.979 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:32:22.979 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:23.287 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:23.290 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:23.301 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":814,"stamp":"2024-09-18T14:32:22.331366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":616856,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":615484,"kb_avail":188126824,"statfs":{"total":193273528320,"available":192641867776,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":630256304},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2608,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1304,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002806"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":15373,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:21.483784+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:21.483784+0000","last_peered":"2024-09-18T14:32:21.483784+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:21.483784+0000","last_undegraded":"2024-09-18T14:32:21.483784+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3576},{"shards":"2,3","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051674,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705982,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.71699999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705981,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306348,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065492,"statfs":{"total":96636764160,"available":96323063808,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.44500000000000001}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.42199999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.39000000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:26.303 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:26.627 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:26.630 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:26.641 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":816,"stamp":"2024-09-18T14:32:26.332311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1828,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":914,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002617"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":15373,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:21.483784+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:21.483784+0000","last_peered":"2024-09-18T14:32:21.483784+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:21.483784+0000","last_undegraded":"2024-09-18T14:32:21.483784+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3576},{"shards":"2,3","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705982,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.71699999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705982,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.434}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:26.642 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:26.954 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:26.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:26.972 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":816,"stamp":"2024-09-18T14:32:26.332311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1828,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":914,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002617"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":15373,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:21.483784+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:21.483784+0000","last_peered":"2024-09-18T14:32:21.483784+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:21.483784+0000","last_undegraded":"2024-09-18T14:32:21.483784+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3576},{"shards":"2,3","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705982,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.71699999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705982,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.434}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:26.973 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:32:26.973 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 15373, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:32:21.483784+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:32:21.483784+0000', 'last_peered': '2024-09-18T14:32:21.483784+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:32:21.483784+0000', 'last_undegraded': '2024-09-18T14:32:21.483784+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 2034, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 3576, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 3576}, {'shards': '2,3', 'objects': 1017}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:32:26.974 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:32:27.375 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:27.376 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1558,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":45,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":1},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635920384,"bytes_avail":192637607936,"bytes_total":193273528320,"misplaced_objects":2034,"misplaced_total":72472,"misplaced_ratio":0.028066011701070755,"recovering_objects_per_sec":76,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":914,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:32:27.390 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:32:27.390 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:27.702 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:27.718 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":816,"stamp":"2024-09-18T14:32:26.332311+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1828,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":914,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002617"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":15373,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:21.483784+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:21.483784+0000","last_peered":"2024-09-18T14:32:21.483784+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:21.483784+0000","last_undegraded":"2024-09-18T14:32:21.483784+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3576,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":3576},{"shards":"2,3","objects":1017}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2034,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34160,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051675,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705982,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46700000000000003,"15min":0.52400000000000002},"min":{"1min":0.29999999999999999,"5min":0.217,"15min":0.20300000000000001},"max":{"1min":0.76800000000000002,"5min":0.97299999999999998,"15min":6.758},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58999999999999997,"15min":0.60799999999999998},"min":{"1min":0.39900000000000002,"5min":0.33400000000000002,"15min":0.24199999999999999},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.61099999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.71699999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705982,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:31:21 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46000000000000002,"15min":0.51300000000000001},"min":{"1min":0.308,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.626,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.57799999999999996,"15min":0.625},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.749,"5min":1.542,"15min":1.542},"last":0.434}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.65400000000000003}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.56100000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:30.720 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:31.004 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:31.006 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:31.021 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":820,"stamp":"2024-09-18T14:32:30.333408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34819,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1318,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002042"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":17350,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:26.486963+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:26.486963+0000","last_peered":"2024-09-18T14:32:26.486963+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:26.486963+0000","last_undegraded":"2024-09-18T14:32:26.486963+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4235,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4235},{"shards":"2,3","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34819,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182493,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705983,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.746}]}]},{"osd":0,"up_from":9,"seq":38654705983,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.47999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.372}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.44},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:31.022 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:31.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:31.310 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:31.322 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":820,"stamp":"2024-09-18T14:32:30.333408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34819,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1318,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002042"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":17350,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:26.486963+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:26.486963+0000","last_peered":"2024-09-18T14:32:26.486963+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:26.486963+0000","last_undegraded":"2024-09-18T14:32:26.486963+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4235,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4235},{"shards":"2,3","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34819,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182493,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705983,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.746}]}]},{"osd":0,"up_from":9,"seq":38654705983,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.47999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.372}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.44},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:31.323 INFO:tasks.osd_recovery.ceph_manager:PG 1.7 is not active+clean 2024-09-18T14:32:31.323 INFO:tasks.osd_recovery.ceph_manager:{'pgid': '1.7', 'version': "17'4593", 'reported_seq': 17350, 'reported_epoch': 45, 'state': 'active+recovering+undersized+remapped', 'last_fresh': '2024-09-18T14:32:26.486963+0000', 'last_change': '2024-09-18T14:31:28.537181+0000', 'last_active': '2024-09-18T14:32:26.486963+0000', 'last_peered': '2024-09-18T14:32:26.486963+0000', 'last_clean': '2024-09-18T14:23:57.130523+0000', 'last_became_active': '2024-09-18T14:24:48.467373+0000', 'last_became_peered': '2024-09-18T14:24:48.467373+0000', 'last_unstale': '2024-09-18T14:32:26.486963+0000', 'last_undegraded': '2024-09-18T14:32:26.486963+0000', 'last_fullsized': '2024-09-18T14:24:48.309745+0000', 'mapping_epoch': 26, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 11, 'last_epoch_clean': 17, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'last_clean_scrub_stamp': '2024-09-18T14:06:38.985939+0000', 'objects_scrubbed': 0, 'log_size': 4593, 'log_dups_size': 0, 'ondisk_log_size': 4593, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 4593, 'num_objects': 4593, 'num_object_clones': 0, 'num_object_copies': 9186, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 716, 'num_objects_unfound': 0, 'num_objects_dirty': 4593, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 4593, 'num_write_kb': 4593, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 4235, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 0], 'acting': [2], 'avail_no_missing': ['2', '3'], 'object_location_counts': [{'shards': '2', 'objects': 4235}, {'shards': '2,3', 'objects': 358}], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 2, 'purged_snaps': []} 2024-09-18T14:32:31.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-18T14:32:31.669 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:31.669 INFO:teuthology.orchestra.run.smithi016.stdout:{"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1 pg undersized","count":1},"muted":false}},"mutes":[]},"election_epoch":6,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1562,"monmap":{"epoch":1,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":47,"num_osds":4,"num_up_osds":4,"osd_up_since":1726669497,"num_in_osds":2,"osd_in_since":1726669441,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":7},{"state_name":"active+recovering+undersized+remapped","count":1}],"num_pgs":8,"num_pools":1,"num_objects":36236,"data_bytes":285461,"bytes_used":635920384,"bytes_avail":192637607936,"bytes_total":193273528320,"misplaced_objects":716,"misplaced_total":72472,"misplaced_ratio":0.0098796776686168458,"recovering_objects_per_sec":82,"recovering_bytes_per_sec":0,"recovering_keys_per_sec":0,"num_objects_recovered":659,"num_bytes_recovered":0,"num_keys_recovered":0},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":0,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":14,"modified":"2024-09-18T14:29:34.287870+0000","services":{}},"progress_events":{}} 2024-09-18T14:32:31.684 INFO:tasks.osd_recovery.ceph_manager:making progress, resetting timeout 2024-09-18T14:32:31.684 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:31.977 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:31.992 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":820,"stamp":"2024-09-18T14:32:30.333408+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34819,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":621016,"kb_used_data":1352,"kb_used_omap":3,"kb_used_meta":619644,"kb_avail":188122664,"statfs":{"total":193273528320,"available":192637607936,"internally_reserved":0,"allocated":1384448,"data_stored":776497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":634516144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-1318,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":659,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002042"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":17350,"reported_epoch":45,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-18T14:32:26.486963+0000","last_change":"2024-09-18T14:31:28.537181+0000","last_active":"2024-09-18T14:32:26.486963+0000","last_peered":"2024-09-18T14:32:26.486963+0000","last_clean":"2024-09-18T14:23:57.130523+0000","last_became_active":"2024-09-18T14:24:48.467373+0000","last_became_peered":"2024-09-18T14:24:48.467373+0000","last_unstale":"2024-09-18T14:32:26.486963+0000","last_undegraded":"2024-09-18T14:32:26.486963+0000","last_fullsized":"2024-09-18T14:24:48.309745+0000","mapping_epoch":26,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:03:30.001575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4235,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[2],"avail_no_missing":["2","3"],"object_location_counts":[{"shards":"2","objects":4235},{"shards":"2,3","objects":358}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4608,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105852+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:31:29.105852+0000","last_peered":"2024-09-18T14:31:29.105852+0000","last_clean":"2024-09-18T14:31:29.105852+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:31:29.105852+0000","last_undegraded":"2024-09-18T14:31:29.105852+0000","last_fullsized":"2024-09-18T14:31:29.105852+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4495,"reported_epoch":45,"state":"active+clean","last_fresh":"2024-09-18T14:31:30.114629+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:31:30.114629+0000","last_peered":"2024-09-18T14:31:30.114629+0000","last_clean":"2024-09-18T14:31:30.114629+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:31:30.114629+0000","last_undegraded":"2024-09-18T14:31:30.114629+0000","last_fullsized":"2024-09-18T14:31:30.114629+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4571,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105684+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:31:29.105684+0000","last_peered":"2024-09-18T14:31:29.105684+0000","last_clean":"2024-09-18T14:31:29.105684+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:31:29.105684+0000","last_undegraded":"2024-09-18T14:31:29.105684+0000","last_fullsized":"2024-09-18T14:31:29.105684+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4534,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109339+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:31:29.109339+0000","last_peered":"2024-09-18T14:31:29.109339+0000","last_clean":"2024-09-18T14:31:29.109339+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:31:29.109339+0000","last_undegraded":"2024-09-18T14:31:29.109339+0000","last_fullsized":"2024-09-18T14:31:29.109339+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4684,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105575+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:31:29.105575+0000","last_peered":"2024-09-18T14:31:29.105575+0000","last_clean":"2024-09-18T14:31:29.105575+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:31:29.105575+0000","last_undegraded":"2024-09-18T14:31:29.105575+0000","last_fullsized":"2024-09-18T14:31:29.105575+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4554,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.105741+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:31:29.105741+0000","last_peered":"2024-09-18T14:31:29.105741+0000","last_clean":"2024-09-18T14:31:29.105741+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:31:29.105741+0000","last_undegraded":"2024-09-18T14:31:29.105741+0000","last_fullsized":"2024-09-18T14:31:29.105741+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4567,"reported_epoch":44,"state":"active+clean","last_fresh":"2024-09-18T14:31:29.109277+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:31:29.109277+0000","last_peered":"2024-09-18T14:31:29.109277+0000","last_clean":"2024-09-18T14:31:29.109277+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:31:29.109277+0000","last_undegraded":"2024-09-18T14:31:29.109277+0000","last_fullsized":"2024-09-18T14:31:29.109277+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":716,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34819,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":15,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051676,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182493,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705983,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.53700000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.45300000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.746}]}]},{"osd":0,"up_from":9,"seq":38654705983,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":310508,"kb_used_data":676,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":94061332,"statfs":{"total":96636764160,"available":96318803968,"internally_reserved":0,"allocated":692224,"data_stored":388119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.47999999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.372}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.44},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.52000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:34.995 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:35.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:35.302 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:35.313 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":822,"stamp":"2024-09-18T14:32:34.334507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":629384,"kb_used_data":1384,"kb_used_omap":3,"kb_used_meta":627964,"kb_avail":188114296,"statfs":{"total":193273528320,"available":192629039104,"internally_reserved":0,"allocated":1417216,"data_stored":789267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":643035824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2034,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002195"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4629,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:29.729655+0000","last_change":"2024-09-18T14:32:29.729655+0000","last_active":"2024-09-18T14:32:29.729655+0000","last_peered":"2024-09-18T14:32:29.729655+0000","last_clean":"2024-09-18T14:32:29.729655+0000","last_became_active":"2024-09-18T14:32:29.725266+0000","last_became_peered":"2024-09-18T14:32:29.725266+0000","last_unstale":"2024-09-18T14:32:29.729655+0000","last_undegraded":"2024-09-18T14:32:29.729655+0000","last_fullsized":"2024-09-18T14:32:29.729655+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:37.280141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714502+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:32:28.714502+0000","last_peered":"2024-09-18T14:32:28.714502+0000","last_clean":"2024-09-18T14:32:28.714502+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:32:28.714502+0000","last_undegraded":"2024-09-18T14:32:28.714502+0000","last_fullsized":"2024-09-18T14:32:28.714502+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4497,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714499+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:32:28.714499+0000","last_peered":"2024-09-18T14:32:28.714499+0000","last_clean":"2024-09-18T14:32:28.714499+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:32:28.714499+0000","last_undegraded":"2024-09-18T14:32:28.714499+0000","last_fullsized":"2024-09-18T14:32:28.714499+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4573,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714641+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:32:28.714641+0000","last_peered":"2024-09-18T14:32:28.714641+0000","last_clean":"2024-09-18T14:32:28.714641+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:32:28.714641+0000","last_undegraded":"2024-09-18T14:32:28.714641+0000","last_fullsized":"2024-09-18T14:32:28.714641+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4536,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717416+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:32:28.717416+0000","last_peered":"2024-09-18T14:32:28.717416+0000","last_clean":"2024-09-18T14:32:28.717416+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:32:28.717416+0000","last_undegraded":"2024-09-18T14:32:28.717416+0000","last_fullsized":"2024-09-18T14:32:28.717416+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4686,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714534+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:32:28.714534+0000","last_peered":"2024-09-18T14:32:28.714534+0000","last_clean":"2024-09-18T14:32:28.714534+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:32:28.714534+0000","last_undegraded":"2024-09-18T14:32:28.714534+0000","last_fullsized":"2024-09-18T14:32:28.714534+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4556,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714420+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:32:28.714420+0000","last_peered":"2024-09-18T14:32:28.714420+0000","last_clean":"2024-09-18T14:32:28.714420+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:32:28.714420+0000","last_undegraded":"2024-09-18T14:32:28.714420+0000","last_fullsized":"2024-09-18T14:32:28.714420+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717427+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:32:28.717427+0000","last_peered":"2024-09-18T14:32:28.717427+0000","last_clean":"2024-09-18T14:32:28.717427+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:32:28.717427+0000","last_undegraded":"2024-09-18T14:32:28.717427+0000","last_fullsized":"2024-09-18T14:32:28.717427+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182494,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73299999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.72299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:35.314 INFO:tasks.osd_recovery.ceph_manager:clean! 2024-09-18T14:32:35.314 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-18T14:32:35.323 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2024-09-18T14:32:35.323 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:32:35.613 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:35.613 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":47,"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","created":"2024-09-18T14:06:29.125568+0000","modified":"2024-09-18T14:32:29.709818+0000","last_up_change":"2024-09-18T14:24:57.391001+0000","last_in_change":"2024-09-18T14:24:01.573778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T14:06:38.348847+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"08356eed-f55f-427f-bce6-f522a976a2c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":22294},{"type":"v1","addr":"172.21.15.16:6809","nonce":22294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":22294},{"type":"v1","addr":"172.21.15.16:6811","nonce":22294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":22294},{"type":"v1","addr":"172.21.15.16:6815","nonce":22294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":22294},{"type":"v1","addr":"172.21.15.16:6813","nonce":22294}]},"public_addr":"172.21.15.16:6809/22294","cluster_addr":"172.21.15.16:6811/22294","heartbeat_back_addr":"172.21.15.16:6815/22294","heartbeat_front_addr":"172.21.15.16:6813/22294","state":["exists","up"]},{"osd":1,"uuid":"851cad7b-f44e-48a3-9d7a-2498617c734e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":22293},{"type":"v1","addr":"172.21.15.16:6825","nonce":22293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":22293},{"type":"v1","addr":"172.21.15.16:6827","nonce":22293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":22293},{"type":"v1","addr":"172.21.15.16:6831","nonce":22293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":22293},{"type":"v1","addr":"172.21.15.16:6829","nonce":22293}]},"public_addr":"172.21.15.16:6825/22293","cluster_addr":"172.21.15.16:6827/22293","heartbeat_back_addr":"172.21.15.16:6831/22293","heartbeat_front_addr":"172.21.15.16:6829/22293","state":["exists","up"]},{"osd":2,"uuid":"8dc73a5a-05ad-471a-8d5b-442d5605ab45","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":21,"up_from":25,"up_thru":45,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":53651},{"type":"v1","addr":"172.21.15.16:6801","nonce":53651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":53651},{"type":"v1","addr":"172.21.15.16:6803","nonce":53651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":53651},{"type":"v1","addr":"172.21.15.16:6807","nonce":53651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":53651},{"type":"v1","addr":"172.21.15.16:6805","nonce":53651}]},"public_addr":"172.21.15.16:6801/53651","cluster_addr":"172.21.15.16:6803/53651","heartbeat_back_addr":"172.21.15.16:6807/53651","heartbeat_front_addr":"172.21.15.16:6805/53651","state":["exists","up"]},{"osd":3,"uuid":"536c2bb9-0d34-4dc1-b8b2-f2df92543320","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":21,"up_from":29,"up_thru":41,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":54496},{"type":"v1","addr":"172.21.15.16:6817","nonce":54496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":54496},{"type":"v1","addr":"172.21.15.16:6819","nonce":54496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":54496},{"type":"v1","addr":"172.21.15.16:6823","nonce":54496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":54496},{"type":"v1","addr":"172.21.15.16:6821","nonce":54496}]},"public_addr":"172.21.15.16:6817/54496","cluster_addr":"172.21.15.16:6819/54496","heartbeat_back_addr":"172.21.15.16:6823/54496","heartbeat_front_addr":"172.21.15.16:6821/54496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.510803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.952904+0000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-18T14:24:32.722152+0000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-18T14:06:35.723520+0000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-18T14:24:32.722152+0000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-18T14:06:35.858647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:32:35.627 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-18T14:32:35.627 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:35.918 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:35.932 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":822,"stamp":"2024-09-18T14:32:34.334507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":629384,"kb_used_data":1384,"kb_used_omap":3,"kb_used_meta":627964,"kb_avail":188114296,"statfs":{"total":193273528320,"available":192629039104,"internally_reserved":0,"allocated":1417216,"data_stored":789267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":643035824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2034,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002195"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4629,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:29.729655+0000","last_change":"2024-09-18T14:32:29.729655+0000","last_active":"2024-09-18T14:32:29.729655+0000","last_peered":"2024-09-18T14:32:29.729655+0000","last_clean":"2024-09-18T14:32:29.729655+0000","last_became_active":"2024-09-18T14:32:29.725266+0000","last_became_peered":"2024-09-18T14:32:29.725266+0000","last_unstale":"2024-09-18T14:32:29.729655+0000","last_undegraded":"2024-09-18T14:32:29.729655+0000","last_fullsized":"2024-09-18T14:32:29.729655+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:37.280141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714502+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:32:28.714502+0000","last_peered":"2024-09-18T14:32:28.714502+0000","last_clean":"2024-09-18T14:32:28.714502+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:32:28.714502+0000","last_undegraded":"2024-09-18T14:32:28.714502+0000","last_fullsized":"2024-09-18T14:32:28.714502+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4497,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714499+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:32:28.714499+0000","last_peered":"2024-09-18T14:32:28.714499+0000","last_clean":"2024-09-18T14:32:28.714499+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:32:28.714499+0000","last_undegraded":"2024-09-18T14:32:28.714499+0000","last_fullsized":"2024-09-18T14:32:28.714499+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4573,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714641+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:32:28.714641+0000","last_peered":"2024-09-18T14:32:28.714641+0000","last_clean":"2024-09-18T14:32:28.714641+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:32:28.714641+0000","last_undegraded":"2024-09-18T14:32:28.714641+0000","last_fullsized":"2024-09-18T14:32:28.714641+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4536,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717416+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:32:28.717416+0000","last_peered":"2024-09-18T14:32:28.717416+0000","last_clean":"2024-09-18T14:32:28.717416+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:32:28.717416+0000","last_undegraded":"2024-09-18T14:32:28.717416+0000","last_fullsized":"2024-09-18T14:32:28.717416+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4686,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714534+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:32:28.714534+0000","last_peered":"2024-09-18T14:32:28.714534+0000","last_clean":"2024-09-18T14:32:28.714534+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:32:28.714534+0000","last_undegraded":"2024-09-18T14:32:28.714534+0000","last_fullsized":"2024-09-18T14:32:28.714534+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4556,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714420+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:32:28.714420+0000","last_peered":"2024-09-18T14:32:28.714420+0000","last_clean":"2024-09-18T14:32:28.714420+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:32:28.714420+0000","last_undegraded":"2024-09-18T14:32:28.714420+0000","last_fullsized":"2024-09-18T14:32:28.714420+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717427+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:32:28.717427+0000","last_peered":"2024-09-18T14:32:28.717427+0000","last_clean":"2024-09-18T14:32:28.717427+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:32:28.717427+0000","last_undegraded":"2024-09-18T14:32:28.717427+0000","last_fullsized":"2024-09-18T14:32:28.717427+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182494,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73299999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.72299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:35.933 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:36.238 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:36.241 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:36.252 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":822,"stamp":"2024-09-18T14:32:34.334507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":629384,"kb_used_data":1384,"kb_used_omap":3,"kb_used_meta":627964,"kb_avail":188114296,"statfs":{"total":193273528320,"available":192629039104,"internally_reserved":0,"allocated":1417216,"data_stored":789267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":643035824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2034,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1017,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002195"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4629,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:29.729655+0000","last_change":"2024-09-18T14:32:29.729655+0000","last_active":"2024-09-18T14:32:29.729655+0000","last_peered":"2024-09-18T14:32:29.729655+0000","last_clean":"2024-09-18T14:32:29.729655+0000","last_became_active":"2024-09-18T14:32:29.725266+0000","last_became_peered":"2024-09-18T14:32:29.725266+0000","last_unstale":"2024-09-18T14:32:29.729655+0000","last_undegraded":"2024-09-18T14:32:29.729655+0000","last_fullsized":"2024-09-18T14:32:29.729655+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:37.280141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714502+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:32:28.714502+0000","last_peered":"2024-09-18T14:32:28.714502+0000","last_clean":"2024-09-18T14:32:28.714502+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:32:28.714502+0000","last_undegraded":"2024-09-18T14:32:28.714502+0000","last_fullsized":"2024-09-18T14:32:28.714502+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4497,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714499+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:32:28.714499+0000","last_peered":"2024-09-18T14:32:28.714499+0000","last_clean":"2024-09-18T14:32:28.714499+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:32:28.714499+0000","last_undegraded":"2024-09-18T14:32:28.714499+0000","last_fullsized":"2024-09-18T14:32:28.714499+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4573,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714641+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:32:28.714641+0000","last_peered":"2024-09-18T14:32:28.714641+0000","last_clean":"2024-09-18T14:32:28.714641+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:32:28.714641+0000","last_undegraded":"2024-09-18T14:32:28.714641+0000","last_fullsized":"2024-09-18T14:32:28.714641+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4536,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717416+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:32:28.717416+0000","last_peered":"2024-09-18T14:32:28.717416+0000","last_clean":"2024-09-18T14:32:28.717416+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:32:28.717416+0000","last_undegraded":"2024-09-18T14:32:28.717416+0000","last_fullsized":"2024-09-18T14:32:28.717416+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4686,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714534+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:32:28.714534+0000","last_peered":"2024-09-18T14:32:28.714534+0000","last_clean":"2024-09-18T14:32:28.714534+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:32:28.714534+0000","last_undegraded":"2024-09-18T14:32:28.714534+0000","last_fullsized":"2024-09-18T14:32:28.714534+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4556,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714420+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:32:28.714420+0000","last_peered":"2024-09-18T14:32:28.714420+0000","last_clean":"2024-09-18T14:32:28.714420+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:32:28.714420+0000","last_undegraded":"2024-09-18T14:32:28.714420+0000","last_fullsized":"2024-09-18T14:32:28.714420+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717427+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:32:28.717427+0000","last_peered":"2024-09-18T14:32:28.717427+0000","last_clean":"2024-09-18T14:32:28.717427+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:32:28.717427+0000","last_undegraded":"2024-09-18T14:32:28.717427+0000","last_fullsized":"2024-09-18T14:32:28.717427+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182494,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73299999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.72299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:36.253 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-18T14:32:36.253 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:36.537 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:36.540 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:36.551 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":823,"stamp":"2024-09-18T14:32:36.334979+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":629384,"kb_used_data":1384,"kb_used_omap":3,"kb_used_meta":627964,"kb_avail":188114296,"statfs":{"total":193273528320,"available":192629039104,"internally_reserved":0,"allocated":1417216,"data_stored":789267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":643035824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-716,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002186"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4629,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:29.729655+0000","last_change":"2024-09-18T14:32:29.729655+0000","last_active":"2024-09-18T14:32:29.729655+0000","last_peered":"2024-09-18T14:32:29.729655+0000","last_clean":"2024-09-18T14:32:29.729655+0000","last_became_active":"2024-09-18T14:32:29.725266+0000","last_became_peered":"2024-09-18T14:32:29.725266+0000","last_unstale":"2024-09-18T14:32:29.729655+0000","last_undegraded":"2024-09-18T14:32:29.729655+0000","last_fullsized":"2024-09-18T14:32:29.729655+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:37.280141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714502+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:32:28.714502+0000","last_peered":"2024-09-18T14:32:28.714502+0000","last_clean":"2024-09-18T14:32:28.714502+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:32:28.714502+0000","last_undegraded":"2024-09-18T14:32:28.714502+0000","last_fullsized":"2024-09-18T14:32:28.714502+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4497,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714499+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:32:28.714499+0000","last_peered":"2024-09-18T14:32:28.714499+0000","last_clean":"2024-09-18T14:32:28.714499+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:32:28.714499+0000","last_undegraded":"2024-09-18T14:32:28.714499+0000","last_fullsized":"2024-09-18T14:32:28.714499+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4573,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714641+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:32:28.714641+0000","last_peered":"2024-09-18T14:32:28.714641+0000","last_clean":"2024-09-18T14:32:28.714641+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:32:28.714641+0000","last_undegraded":"2024-09-18T14:32:28.714641+0000","last_fullsized":"2024-09-18T14:32:28.714641+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4536,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717416+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:32:28.717416+0000","last_peered":"2024-09-18T14:32:28.717416+0000","last_clean":"2024-09-18T14:32:28.717416+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:32:28.717416+0000","last_undegraded":"2024-09-18T14:32:28.717416+0000","last_fullsized":"2024-09-18T14:32:28.717416+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4686,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714534+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:32:28.714534+0000","last_peered":"2024-09-18T14:32:28.714534+0000","last_clean":"2024-09-18T14:32:28.714534+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:32:28.714534+0000","last_undegraded":"2024-09-18T14:32:28.714534+0000","last_fullsized":"2024-09-18T14:32:28.714534+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4556,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714420+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:32:28.714420+0000","last_peered":"2024-09-18T14:32:28.714420+0000","last_clean":"2024-09-18T14:32:28.714420+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:32:28.714420+0000","last_undegraded":"2024-09-18T14:32:28.714420+0000","last_fullsized":"2024-09-18T14:32:28.714420+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717427+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:32:28.717427+0000","last_peered":"2024-09-18T14:32:28.717427+0000","last_clean":"2024-09-18T14:32:28.717427+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:32:28.717427+0000","last_undegraded":"2024-09-18T14:32:28.717427+0000","last_fullsized":"2024-09-18T14:32:28.717427+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1048576,"data_stored":813364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051677,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182494,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.5}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.75800000000000001}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.73299999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705984,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.66400000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.58499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.72299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:36.552 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-18T14:32:36.844 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:36.845 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":47,"fsid":"946c9667-3cd7-48a9-8c9c-fd6b4af766c9","created":"2024-09-18T14:06:29.125568+0000","modified":"2024-09-18T14:32:29.709818+0000","last_up_change":"2024-09-18T14:24:57.391001+0000","last_in_change":"2024-09-18T14:24:01.573778+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":4,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2024-09-18T14:06:38.348847+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":14,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"08356eed-f55f-427f-bce6-f522a976a2c9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":22294},{"type":"v1","addr":"172.21.15.16:6809","nonce":22294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":22294},{"type":"v1","addr":"172.21.15.16:6811","nonce":22294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6814","nonce":22294},{"type":"v1","addr":"172.21.15.16:6815","nonce":22294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":22294},{"type":"v1","addr":"172.21.15.16:6813","nonce":22294}]},"public_addr":"172.21.15.16:6809/22294","cluster_addr":"172.21.15.16:6811/22294","heartbeat_back_addr":"172.21.15.16:6815/22294","heartbeat_front_addr":"172.21.15.16:6813/22294","state":["exists","up"]},{"osd":1,"uuid":"851cad7b-f44e-48a3-9d7a-2498617c734e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6824","nonce":22293},{"type":"v1","addr":"172.21.15.16:6825","nonce":22293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":22293},{"type":"v1","addr":"172.21.15.16:6827","nonce":22293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":22293},{"type":"v1","addr":"172.21.15.16:6831","nonce":22293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6828","nonce":22293},{"type":"v1","addr":"172.21.15.16:6829","nonce":22293}]},"public_addr":"172.21.15.16:6825/22293","cluster_addr":"172.21.15.16:6827/22293","heartbeat_back_addr":"172.21.15.16:6831/22293","heartbeat_front_addr":"172.21.15.16:6829/22293","state":["exists","up"]},{"osd":2,"uuid":"8dc73a5a-05ad-471a-8d5b-442d5605ab45","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":21,"up_from":25,"up_thru":45,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":53651},{"type":"v1","addr":"172.21.15.16:6801","nonce":53651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":53651},{"type":"v1","addr":"172.21.15.16:6803","nonce":53651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":53651},{"type":"v1","addr":"172.21.15.16:6807","nonce":53651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":53651},{"type":"v1","addr":"172.21.15.16:6805","nonce":53651}]},"public_addr":"172.21.15.16:6801/53651","cluster_addr":"172.21.15.16:6803/53651","heartbeat_back_addr":"172.21.15.16:6807/53651","heartbeat_front_addr":"172.21.15.16:6805/53651","state":["exists","up"]},{"osd":3,"uuid":"536c2bb9-0d34-4dc1-b8b2-f2df92543320","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":9,"last_clean_end":21,"up_from":29,"up_thru":41,"down_at":22,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":54496},{"type":"v1","addr":"172.21.15.16:6817","nonce":54496}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":54496},{"type":"v1","addr":"172.21.15.16:6819","nonce":54496}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":54496},{"type":"v1","addr":"172.21.15.16:6823","nonce":54496}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6820","nonce":54496},{"type":"v1","addr":"172.21.15.16:6821","nonce":54496}]},"public_addr":"172.21.15.16:6817/54496","cluster_addr":"172.21.15.16:6819/54496","heartbeat_back_addr":"172.21.15.16:6823/54496","heartbeat_front_addr":"172.21.15.16:6821/54496","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.510803+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-18T14:06:35.952904+0000","dead_epoch":0},{"osd":2,"down_stamp":"2024-09-18T14:24:32.722152+0000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-18T14:06:35.723520+0000","dead_epoch":0},{"osd":3,"down_stamp":"2024-09-18T14:24:32.722152+0000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":65536,"last_purged_snaps_scrub":"2024-09-18T14:06:35.858647+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-18T14:32:37.859 INFO:tasks.ceph:Scrubbing osd.0 2024-09-18T14:32:37.860 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:32:38.013 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:32:38.013 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:32:38.013 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:32:38.026 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-18T14:32:38.333 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 0 to deep-scrub 2024-09-18T14:32:38.343 INFO:tasks.ceph:Scrubbing osd.1 2024-09-18T14:32:38.344 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:32:38.484 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:32:38.484 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:32:38.485 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:32:38.496 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-18T14:32:38.791 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 1 to deep-scrub 2024-09-18T14:32:38.802 INFO:tasks.ceph:Scrubbing osd.2 2024-09-18T14:32:38.802 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:32:38.941 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:32:38.941 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:32:38.941 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:32:38.953 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-18T14:32:39.262 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 2 to deep-scrub 2024-09-18T14:32:39.277 INFO:tasks.ceph:Scrubbing osd.3 2024-09-18T14:32:39.277 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-18T14:32:39.421 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2024-09-18T14:32:39.422 INFO:teuthology.orchestra.run.smithi016.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-18T14:32:39.422 INFO:teuthology.orchestra.run.smithi016.stdout:} 2024-09-18T14:32:39.434 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-18T14:32:39.789 INFO:teuthology.orchestra.run.smithi016.stderr:instructed osd(s) 3 to deep-scrub 2024-09-18T14:32:39.801 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:32:40.144 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:32:40.147 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:32:40.160 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":824,"stamp":"2024-09-18T14:32:38.335335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":629384,"kb_used_data":1384,"kb_used_omap":3,"kb_used_meta":627964,"kb_avail":188114296,"statfs":{"total":193273528320,"available":192629039104,"internally_reserved":0,"allocated":1417216,"data_stored":789267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":643035824},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-716,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":358,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.625422"},"pg_stats":[{"pgid":"1.7","version":"17'4593","reported_seq":4629,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:29.729655+0000","last_change":"2024-09-18T14:32:29.729655+0000","last_active":"2024-09-18T14:32:29.729655+0000","last_peered":"2024-09-18T14:32:29.729655+0000","last_clean":"2024-09-18T14:32:29.729655+0000","last_became_active":"2024-09-18T14:32:29.725266+0000","last_became_peered":"2024-09-18T14:32:29.725266+0000","last_unstale":"2024-09-18T14:32:29.729655+0000","last_undegraded":"2024-09-18T14:32:29.729655+0000","last_fullsized":"2024-09-18T14:32:29.729655+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4593,"log_dups_size":0,"ondisk_log_size":4593,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:42:37.280141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"17'4548","reported_seq":4610,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714502+0000","last_change":"2024-09-18T14:30:34.719268+0000","last_active":"2024-09-18T14:32:28.714502+0000","last_peered":"2024-09-18T14:32:28.714502+0000","last_clean":"2024-09-18T14:32:28.714502+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:32:28.714502+0000","last_undegraded":"2024-09-18T14:32:28.714502+0000","last_fullsized":"2024-09-18T14:32:28.714502+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4548,"log_dups_size":0,"ondisk_log_size":4548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:25:10.083105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4497,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714499+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:32:28.714499+0000","last_peered":"2024-09-18T14:32:28.714499+0000","last_clean":"2024-09-18T14:32:28.714499+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:32:28.714499+0000","last_undegraded":"2024-09-18T14:32:28.714499+0000","last_fullsized":"2024-09-18T14:32:28.714499+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:02:33.689207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4573,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714641+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:32:28.714641+0000","last_peered":"2024-09-18T14:32:28.714641+0000","last_clean":"2024-09-18T14:32:28.714641+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:32:28.714641+0000","last_undegraded":"2024-09-18T14:32:28.714641+0000","last_fullsized":"2024-09-18T14:32:28.714641+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:02:54.734543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"17'4482","reported_seq":4536,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717416+0000","last_change":"2024-09-18T14:27:39.555275+0000","last_active":"2024-09-18T14:32:28.717416+0000","last_peered":"2024-09-18T14:32:28.717416+0000","last_clean":"2024-09-18T14:32:28.717416+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:32:28.717416+0000","last_undegraded":"2024-09-18T14:32:28.717416+0000","last_fullsized":"2024-09-18T14:32:28.717416+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4482,"log_dups_size":0,"ondisk_log_size":4482,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:19:56.212826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4686,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714534+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:32:28.714534+0000","last_peered":"2024-09-18T14:32:28.714534+0000","last_clean":"2024-09-18T14:32:28.714534+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:32:28.714534+0000","last_undegraded":"2024-09-18T14:32:28.714534+0000","last_fullsized":"2024-09-18T14:32:28.714534+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T21:24:46.587784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"17'4490","reported_seq":4556,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.714420+0000","last_change":"2024-09-18T14:28:37.098807+0000","last_active":"2024-09-18T14:32:28.714420+0000","last_peered":"2024-09-18T14:32:28.714420+0000","last_clean":"2024-09-18T14:32:28.714420+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:32:28.714420+0000","last_undegraded":"2024-09-18T14:32:28.714420+0000","last_fullsized":"2024-09-18T14:32:28.714420+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4490,"log_dups_size":0,"ondisk_log_size":4490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:10:09.773014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'4531","reported_seq":4569,"reported_epoch":46,"state":"active+clean","last_fresh":"2024-09-18T14:32:28.717427+0000","last_change":"2024-09-18T14:29:35.501274+0000","last_active":"2024-09-18T14:32:28.717427+0000","last_peered":"2024-09-18T14:32:28.717427+0000","last_clean":"2024-09-18T14:32:28.717427+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:32:28.717427+0000","last_undegraded":"2024-09-18T14:32:28.717427+0000","last_fullsized":"2024-09-18T14:32:28.717427+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4531,"log_dups_size":0,"ondisk_log_size":4531,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:37.820405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1015808,"data_stored":787132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":36237,"ondisk_log_size":36237,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051678,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182495,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705985,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.72799999999999998}]},{"osd":2,"last update":"Wed Sep 18 14:31:52 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.69799999999999995,"5min":1.357,"15min":1.357},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56100000000000005,"15min":2.0960000000000001},"min":{"1min":0.36599999999999999,"5min":0.34899999999999998,"15min":0.26600000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":46.807000000000002},"last":0.60099999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.754}]}]},{"osd":0,"up_from":9,"seq":38654705985,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":314692,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":94057148,"statfs":{"total":96636764160,"available":96314519552,"internally_reserved":0,"allocated":708608,"data_stored":394504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.81899999999999995}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.748}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.88100000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":223015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":65623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:32:40.164 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:32:40.164 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:32:40.164 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:32:40.164 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:32:40.165 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:32:40.165 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:32:40.165 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:32:40.165 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:32:40.165 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T14:33:00.166 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:33:00.629 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:33:00.639 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:33:00.647 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":835,"stamp":"2024-09-18T14:33:00.339339+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51013,"ondisk_log_size":51013,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596088,"kb_used_data":1384,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147592,"statfs":{"total":193273528320,"available":192663134208,"internally_reserved":0,"allocated":1417216,"data_stored":789267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1,3,8,5,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002278"},"pg_stats":[{"pgid":"1.7","version":"47'5542","reported_seq":5622,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-18T14:32:57.455081+0000","last_change":"2024-09-18T14:32:55.307203+0000","last_active":"2024-09-18T14:32:57.455081+0000","last_peered":"2024-09-18T14:32:57.455081+0000","last_clean":"2024-09-18T14:32:57.455081+0000","last_became_active":"2024-09-18T14:32:29.725266+0000","last_became_peered":"2024-09-18T14:32:29.725266+0000","last_unstale":"2024-09-18T14:32:57.455081+0000","last_undegraded":"2024-09-18T14:32:57.455081+0000","last_fullsized":"2024-09-18T14:32:57.455081+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":974,"log_size":5542,"log_dups_size":0,"ondisk_log_size":5542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 2s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'4872","reported_seq":4941,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-18T14:32:58.185005+0000","last_change":"2024-09-18T14:32:57.597222+0000","last_active":"2024-09-18T14:32:58.185005+0000","last_peered":"2024-09-18T14:32:58.185005+0000","last_clean":"2024-09-18T14:32:58.185005+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:32:58.185005+0000","last_undegraded":"2024-09-18T14:32:58.185005+0000","last_fullsized":"2024-09-18T14:32:58.185005+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":324,"log_size":4872,"log_dups_size":0,"ondisk_log_size":4872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 0s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"17'4453","reported_seq":4498,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:29.724372+0000","last_change":"2024-09-18T14:31:30.114629+0000","last_active":"2024-09-18T14:32:29.724372+0000","last_peered":"2024-09-18T14:32:29.724372+0000","last_clean":"2024-09-18T14:32:29.724372+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:32:29.724372+0000","last_undegraded":"2024-09-18T14:32:29.724372+0000","last_fullsized":"2024-09-18T14:32:29.724372+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4453,"log_dups_size":0,"ondisk_log_size":4453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"17'4503","reported_seq":4574,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:29.724350+0000","last_change":"2024-09-18T14:26:43.338226+0000","last_active":"2024-09-18T14:32:29.724350+0000","last_peered":"2024-09-18T14:32:29.724350+0000","last_clean":"2024-09-18T14:32:29.724350+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:32:29.724350+0000","last_undegraded":"2024-09-18T14:32:29.724350+0000","last_fullsized":"2024-09-18T14:32:29.724350+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4503,"log_dups_size":0,"ondisk_log_size":4503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"47'8964","reported_seq":9208,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:56.777484+0000","last_change":"2024-09-18T14:32:56.777457+0000","last_active":"2024-09-18T14:32:56.777484+0000","last_peered":"2024-09-18T14:32:56.777484+0000","last_clean":"2024-09-18T14:32:56.777484+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:32:56.777484+0000","last_undegraded":"2024-09-18T14:32:56.777484+0000","last_fullsized":"2024-09-18T14:32:56.777484+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8964","last_scrub_stamp":"2024-09-18T14:32:56.777338+0000","last_deep_scrub":"47'8964","last_deep_scrub_stamp":"2024-09-18T14:32:56.777338+0000","last_clean_scrub_stamp":"2024-09-18T14:32:56.777338+0000","objects_scrubbed":4482,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:26:11.615377+0000","scrub_duration":15.348717383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"17'4637","reported_seq":4687,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:29.724395+0000","last_change":"2024-09-18T14:25:50.065230+0000","last_active":"2024-09-18T14:32:29.724395+0000","last_peered":"2024-09-18T14:32:29.724395+0000","last_clean":"2024-09-18T14:32:29.724395+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:32:29.724395+0000","last_undegraded":"2024-09-18T14:32:29.724395+0000","last_fullsized":"2024-09-18T14:32:29.724395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":0,"log_size":4637,"log_dups_size":0,"ondisk_log_size":4637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"47'8980","reported_seq":9236,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:57.037673+0000","last_change":"2024-09-18T14:32:57.037629+0000","last_active":"2024-09-18T14:32:57.037673+0000","last_peered":"2024-09-18T14:32:57.037673+0000","last_clean":"2024-09-18T14:32:57.037673+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:32:57.037673+0000","last_undegraded":"2024-09-18T14:32:57.037673+0000","last_fullsized":"2024-09-18T14:32:57.037673+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8980","last_scrub_stamp":"2024-09-18T14:32:57.037498+0000","last_deep_scrub":"47'8980","last_deep_scrub_stamp":"2024-09-18T14:32:57.037498+0000","last_clean_scrub_stamp":"2024-09-18T14:32:57.037498+0000","objects_scrubbed":4490,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:56:35.060495+0000","scrub_duration":15.365038688,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"47'9062","reported_seq":9292,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:54.591111+0000","last_change":"2024-09-18T14:32:54.591048+0000","last_active":"2024-09-18T14:32:54.591111+0000","last_peered":"2024-09-18T14:32:54.591111+0000","last_clean":"2024-09-18T14:32:54.591111+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:32:54.591111+0000","last_undegraded":"2024-09-18T14:32:54.591111+0000","last_fullsized":"2024-09-18T14:32:54.591111+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9062","last_scrub_stamp":"2024-09-18T14:32:54.590936+0000","last_deep_scrub":"47'9062","last_deep_scrub_stamp":"2024-09-18T14:32:54.590936+0000","last_clean_scrub_stamp":"2024-09-18T14:32:54.590936+0000","objects_scrubbed":4531,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:46:39.935295+0000","scrub_duration":15.146936148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":897024,"data_stored":695301,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":51013,"ondisk_log_size":51013,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051682,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182499,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298044,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073796,"statfs":{"total":96636764160,"available":96331567104,"internally_reserved":0,"allocated":708608,"data_stored":394763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,0,1,5,5],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.28299999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.48899999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.79300000000000004,"5min":1.357,"15min":1.357},"last":0.373},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55700000000000005,"15min":1.9239999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.26600000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":46.807000000000002},"last":0.47799999999999998}]},{"osd":3,"last update":"Wed Sep 18 14:32:15 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55000000000000004,"15min":0.54100000000000004},"min":{"1min":0.38600000000000001,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.748,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.67700000000000005,"15min":0.64000000000000001},"min":{"1min":0.42199999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.38600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705989,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298044,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073796,"statfs":{"total":96636764160,"available":96331567104,"internally_reserved":0,"allocated":708608,"data_stored":394504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1,2,3,0,2],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.28000000000000003}]},{"osd":2,"last update":"Wed Sep 18 14:32:01 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.29899999999999999,"15min":0.26300000000000001},"max":{"1min":0.81299999999999994,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.52300000000000002,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.28399999999999997,"15min":0.247},"max":{"1min":0.75800000000000001,"5min":0.96199999999999997,"15min":3.1920000000000002},"last":0.46999999999999997}]},{"osd":3,"last update":"Wed Sep 18 14:32:13 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59499999999999997,"15min":0.61199999999999999},"min":{"1min":0.38200000000000001,"5min":0.38,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.98799999999999999,"15min":1.0580000000000001},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.39000000000000001,"5min":0.36099999999999999,"15min":0.317},"max":{"1min":0.81000000000000005,"5min":1.008,"15min":1.008},"last":0.36699999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":253952,"data_stored":196783,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:33:00.648 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:33:00.648 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:33:00.648 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:33:00.648 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:33:00.649 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:33:00.649 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T14:33:20.650 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:33:21.058 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:33:21.061 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:33:21.074 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":845,"stamp":"2024-09-18T14:33:20.342886+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67132,"ondisk_log_size":67132,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":612728,"kb_used_data":1384,"kb_used_omap":3,"kb_used_meta":611324,"kb_avail":188130952,"statfs":{"total":193273528320,"available":192646094848,"internally_reserved":0,"allocated":1417216,"data_stored":789267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":625996464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002172"},"pg_stats":[{"pgid":"1.7","version":"47'9186","reported_seq":9416,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:33:10.771492+0000","last_change":"2024-09-18T14:33:10.771457+0000","last_active":"2024-09-18T14:33:10.771492+0000","last_peered":"2024-09-18T14:33:10.771492+0000","last_clean":"2024-09-18T14:33:10.771492+0000","last_became_active":"2024-09-18T14:32:29.725266+0000","last_became_peered":"2024-09-18T14:32:29.725266+0000","last_unstale":"2024-09-18T14:33:10.771492+0000","last_undegraded":"2024-09-18T14:33:10.771492+0000","last_fullsized":"2024-09-18T14:33:10.771492+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9186","last_scrub_stamp":"2024-09-18T14:33:10.771339+0000","last_deep_scrub":"47'9186","last_deep_scrub_stamp":"2024-09-18T14:33:10.771339+0000","last_clean_scrub_stamp":"2024-09-18T14:33:10.771339+0000","objects_scrubbed":4593,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:14:52.331487+0000","scrub_duration":15.463919324000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'9096","reported_seq":9350,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:33:12.380449+0000","last_change":"2024-09-18T14:33:12.380423+0000","last_active":"2024-09-18T14:33:12.380449+0000","last_peered":"2024-09-18T14:33:12.380449+0000","last_clean":"2024-09-18T14:33:12.380449+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:33:12.380449+0000","last_undegraded":"2024-09-18T14:33:12.380449+0000","last_fullsized":"2024-09-18T14:33:12.380449+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9096","last_scrub_stamp":"2024-09-18T14:33:12.380329+0000","last_deep_scrub":"47'9096","last_deep_scrub_stamp":"2024-09-18T14:33:12.380329+0000","last_clean_scrub_stamp":"2024-09-18T14:33:12.380329+0000","objects_scrubbed":4548,"log_size":9096,"log_dups_size":0,"ondisk_log_size":9096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:55:19.676396+0000","scrub_duration":14.782908825,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"47'6152","reported_seq":6270,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-18T14:33:18.353141+0000","last_change":"2024-09-18T14:33:14.655783+0000","last_active":"2024-09-18T14:33:18.353141+0000","last_peered":"2024-09-18T14:33:18.353141+0000","last_clean":"2024-09-18T14:33:18.353141+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:33:18.353141+0000","last_undegraded":"2024-09-18T14:33:18.353141+0000","last_fullsized":"2024-09-18T14:33:18.353141+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":1724,"log_size":6152,"log_dups_size":0,"ondisk_log_size":6152,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 3s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"47'9006","reported_seq":9267,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:33:13.055770+0000","last_change":"2024-09-18T14:33:13.055720+0000","last_active":"2024-09-18T14:33:13.055770+0000","last_peered":"2024-09-18T14:33:13.055770+0000","last_clean":"2024-09-18T14:33:13.055770+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:33:13.055770+0000","last_undegraded":"2024-09-18T14:33:13.055770+0000","last_fullsized":"2024-09-18T14:33:13.055770+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9006","last_scrub_stamp":"2024-09-18T14:33:13.055575+0000","last_deep_scrub":"47'9006","last_deep_scrub_stamp":"2024-09-18T14:33:13.055575+0000","last_clean_scrub_stamp":"2024-09-18T14:33:13.055575+0000","objects_scrubbed":4503,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:21:09.064859+0000","scrub_duration":14.464990638,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"47'8964","reported_seq":9208,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:56.777484+0000","last_change":"2024-09-18T14:32:56.777457+0000","last_active":"2024-09-18T14:32:56.777484+0000","last_peered":"2024-09-18T14:32:56.777484+0000","last_clean":"2024-09-18T14:32:56.777484+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:32:56.777484+0000","last_undegraded":"2024-09-18T14:32:56.777484+0000","last_fullsized":"2024-09-18T14:32:56.777484+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8964","last_scrub_stamp":"2024-09-18T14:32:56.777338+0000","last_deep_scrub":"47'8964","last_deep_scrub_stamp":"2024-09-18T14:32:56.777338+0000","last_clean_scrub_stamp":"2024-09-18T14:32:56.777338+0000","objects_scrubbed":4482,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:26:11.615377+0000","scrub_duration":15.348717383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'6686","reported_seq":6823,"reported_epoch":47,"state":"active+clean+scrubbing+deep","last_fresh":"2024-09-18T14:33:18.382609+0000","last_change":"2024-09-18T14:33:13.660257+0000","last_active":"2024-09-18T14:33:18.382609+0000","last_peered":"2024-09-18T14:33:18.382609+0000","last_clean":"2024-09-18T14:33:18.382609+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:33:18.382609+0000","last_undegraded":"2024-09-18T14:33:18.382609+0000","last_fullsized":"2024-09-18T14:33:18.382609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-18T14:06:38.985939+0000","last_clean_scrub_stamp":"2024-09-18T14:06:38.985939+0000","objects_scrubbed":2074,"log_size":6686,"log_dups_size":0,"ondisk_log_size":6686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"deep scrubbing for 4s","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"47'8980","reported_seq":9236,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:57.037673+0000","last_change":"2024-09-18T14:32:57.037629+0000","last_active":"2024-09-18T14:32:57.037673+0000","last_peered":"2024-09-18T14:32:57.037673+0000","last_clean":"2024-09-18T14:32:57.037673+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:32:57.037673+0000","last_undegraded":"2024-09-18T14:32:57.037673+0000","last_fullsized":"2024-09-18T14:32:57.037673+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8980","last_scrub_stamp":"2024-09-18T14:32:57.037498+0000","last_deep_scrub":"47'8980","last_deep_scrub_stamp":"2024-09-18T14:32:57.037498+0000","last_clean_scrub_stamp":"2024-09-18T14:32:57.037498+0000","objects_scrubbed":4490,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:56:35.060495+0000","scrub_duration":15.365038688,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"47'9062","reported_seq":9292,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:54.591111+0000","last_change":"2024-09-18T14:32:54.591048+0000","last_active":"2024-09-18T14:32:54.591111+0000","last_peered":"2024-09-18T14:32:54.591111+0000","last_clean":"2024-09-18T14:32:54.591111+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:32:54.591111+0000","last_undegraded":"2024-09-18T14:32:54.591111+0000","last_fullsized":"2024-09-18T14:32:54.591111+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9062","last_scrub_stamp":"2024-09-18T14:32:54.590936+0000","last_deep_scrub":"47'9062","last_deep_scrub_stamp":"2024-09-18T14:32:54.590936+0000","last_clean_scrub_stamp":"2024-09-18T14:32:54.590936+0000","objects_scrubbed":4531,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:46:39.935295+0000","scrub_duration":15.146936148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":847872,"data_stored":655953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67132,"ondisk_log_size":67132,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051686,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182503,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705993,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306364,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065476,"statfs":{"total":96636764160,"available":96323047424,"internally_reserved":0,"allocated":708608,"data_stored":394763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,2,2,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.46400000000000002,"15min":0.51900000000000002},"min":{"1min":0.27400000000000002,"5min":0.26900000000000002,"15min":0.20300000000000001},"max":{"1min":0.68600000000000005,"5min":0.97299999999999998,"15min":6.758},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.58899999999999997,"15min":0.60399999999999998},"min":{"1min":0.38200000000000001,"5min":0.36499999999999999,"15min":0.24199999999999999},"max":{"1min":0.82599999999999996,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.55200000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.48899999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.79300000000000004,"5min":1.357,"15min":1.357},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55700000000000005,"15min":1.9239999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.26600000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":46.807000000000002},"last":0.58399999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.72999999999999998,"5min":0.81399999999999995,"15min":1.0309999999999999},"last":0.501},{"interface":"front","average":{"1min":0.628,"5min":0.65600000000000003,"15min":0.64100000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92000000000000004,"5min":0.96399999999999997,"15min":1.1339999999999999},"last":0.52900000000000003}]}]},{"osd":0,"up_from":9,"seq":38654705993,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306364,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":94065476,"statfs":{"total":96636764160,"available":96323047424,"internally_reserved":0,"allocated":708608,"data_stored":394504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.51100000000000001},"min":{"1min":0.32000000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.67600000000000005,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.56499999999999995,"15min":0.61799999999999999},"min":{"1min":0.434,"5min":0.31900000000000001,"15min":0.26500000000000001},"max":{"1min":0.92100000000000004,"5min":1.542,"15min":1.542},"last":0.34499999999999997}]},{"osd":2,"last update":"Wed Sep 18 14:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.80000000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.247},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":3.1920000000000002},"last":0.37}]},{"osd":3,"last update":"Wed Sep 18 14:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.61399999999999999,"15min":0.61699999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":2.895,"5min":2.895,"15min":2.895},"last":0.441},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59799999999999998,"15min":0.61099999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.317},"max":{"1min":0.88100000000000001,"5min":1.008,"15min":1.008},"last":0.64300000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":157435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":24,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:33:21.075 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:33:21.075 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2024-09-18T14:06:38.985939+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=6, tm_sec=38, tm_wday=2, tm_yday=262, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=18, tm_hour=14, tm_min=32, tm_sec=36, tm_wday=2, tm_yday=262, tm_isdst=0) 2024-09-18T14:33:21.075 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-18T14:33:41.076 DEBUG:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-18T14:33:41.399 INFO:teuthology.orchestra.run.smithi016.stdout: 2024-09-18T14:33:41.402 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2024-09-18T14:33:41.412 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":855,"stamp":"2024-09-18T14:33:40.348443+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72473,"ondisk_log_size":72473,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":596088,"kb_used_data":1384,"kb_used_omap":3,"kb_used_meta":594684,"kb_avail":188147592,"statfs":{"total":193273528320,"available":192663134208,"internally_reserved":0,"allocated":1417216,"data_stored":789267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3408,"internal_metadata":608957104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004175"},"pg_stats":[{"pgid":"1.7","version":"47'9186","reported_seq":9416,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:33:10.771492+0000","last_change":"2024-09-18T14:33:10.771457+0000","last_active":"2024-09-18T14:33:10.771492+0000","last_peered":"2024-09-18T14:33:10.771492+0000","last_clean":"2024-09-18T14:33:10.771492+0000","last_became_active":"2024-09-18T14:32:29.725266+0000","last_became_peered":"2024-09-18T14:32:29.725266+0000","last_unstale":"2024-09-18T14:33:10.771492+0000","last_undegraded":"2024-09-18T14:33:10.771492+0000","last_fullsized":"2024-09-18T14:33:10.771492+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9186","last_scrub_stamp":"2024-09-18T14:33:10.771339+0000","last_deep_scrub":"47'9186","last_deep_scrub_stamp":"2024-09-18T14:33:10.771339+0000","last_clean_scrub_stamp":"2024-09-18T14:33:10.771339+0000","objects_scrubbed":4593,"log_size":9186,"log_dups_size":0,"ondisk_log_size":9186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T16:14:52.331487+0000","scrub_duration":15.463919324000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4593,"num_objects":4593,"num_object_clones":0,"num_object_copies":9186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4593,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4593,"num_write_kb":4593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4593,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"47'9096","reported_seq":9350,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:33:12.380449+0000","last_change":"2024-09-18T14:33:12.380423+0000","last_active":"2024-09-18T14:33:12.380449+0000","last_peered":"2024-09-18T14:33:12.380449+0000","last_clean":"2024-09-18T14:33:12.380449+0000","last_became_active":"2024-09-18T14:30:34.714211+0000","last_became_peered":"2024-09-18T14:30:34.714211+0000","last_unstale":"2024-09-18T14:33:12.380449+0000","last_undegraded":"2024-09-18T14:33:12.380449+0000","last_fullsized":"2024-09-18T14:33:12.380449+0000","mapping_epoch":42,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":43,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9096","last_scrub_stamp":"2024-09-18T14:33:12.380329+0000","last_deep_scrub":"47'9096","last_deep_scrub_stamp":"2024-09-18T14:33:12.380329+0000","last_clean_scrub_stamp":"2024-09-18T14:33:12.380329+0000","objects_scrubbed":4548,"log_size":9096,"log_dups_size":0,"ondisk_log_size":9096,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:55:19.676396+0000","scrub_duration":14.782908825,"objects_trimmed":0,"snaptrim_duration":0.00097209900000000003,"stat_sum":{"num_bytes":57008,"num_objects":4548,"num_object_clones":0,"num_object_copies":9096,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":4596,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4548,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.5","version":"47'8906","reported_seq":9139,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:33:27.159025+0000","last_change":"2024-09-18T14:33:27.158974+0000","last_active":"2024-09-18T14:33:27.159025+0000","last_peered":"2024-09-18T14:33:27.159025+0000","last_clean":"2024-09-18T14:33:27.159025+0000","last_became_active":"2024-09-18T14:31:30.111108+0000","last_became_peered":"2024-09-18T14:31:30.111108+0000","last_unstale":"2024-09-18T14:33:27.159025+0000","last_undegraded":"2024-09-18T14:33:27.159025+0000","last_fullsized":"2024-09-18T14:33:27.159025+0000","mapping_epoch":44,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8906","last_scrub_stamp":"2024-09-18T14:33:27.158796+0000","last_deep_scrub":"47'8906","last_deep_scrub_stamp":"2024-09-18T14:33:27.158796+0000","last_clean_scrub_stamp":"2024-09-18T14:33:27.158796+0000","objects_scrubbed":4453,"log_size":8906,"log_dups_size":0,"ondisk_log_size":8906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":13,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:13:17.358176+0000","scrub_duration":12.502771965000001,"objects_trimmed":0,"snaptrim_duration":0.00074048799999999998,"stat_sum":{"num_bytes":30706,"num_objects":4453,"num_object_clones":0,"num_object_copies":8906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4453,"num_write_kb":4477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4453,"num_bytes_recovered":26256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"47'9006","reported_seq":9267,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:33:13.055770+0000","last_change":"2024-09-18T14:33:13.055720+0000","last_active":"2024-09-18T14:33:13.055770+0000","last_peered":"2024-09-18T14:33:13.055770+0000","last_clean":"2024-09-18T14:33:13.055770+0000","last_became_active":"2024-09-18T14:26:43.326224+0000","last_became_peered":"2024-09-18T14:26:43.326224+0000","last_unstale":"2024-09-18T14:33:13.055770+0000","last_undegraded":"2024-09-18T14:33:13.055770+0000","last_fullsized":"2024-09-18T14:33:13.055770+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9006","last_scrub_stamp":"2024-09-18T14:33:13.055575+0000","last_deep_scrub":"47'9006","last_deep_scrub_stamp":"2024-09-18T14:33:13.055575+0000","last_clean_scrub_stamp":"2024-09-18T14:33:13.055575+0000","objects_scrubbed":4503,"log_size":9006,"log_dups_size":0,"ondisk_log_size":9006,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":15,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T22:21:09.064859+0000","scrub_duration":14.464990638,"objects_trimmed":0,"snaptrim_duration":0.00065800600000000004,"stat_sum":{"num_bytes":56963,"num_objects":4503,"num_object_clones":0,"num_object_copies":9006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4503,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4503,"num_write_kb":4551,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3442,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"47'8964","reported_seq":9208,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:56.777484+0000","last_change":"2024-09-18T14:32:56.777457+0000","last_active":"2024-09-18T14:32:56.777484+0000","last_peered":"2024-09-18T14:32:56.777484+0000","last_clean":"2024-09-18T14:32:56.777484+0000","last_became_active":"2024-09-18T14:27:39.550223+0000","last_became_peered":"2024-09-18T14:27:39.550223+0000","last_unstale":"2024-09-18T14:32:56.777484+0000","last_undegraded":"2024-09-18T14:32:56.777484+0000","last_fullsized":"2024-09-18T14:32:56.777484+0000","mapping_epoch":36,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":37,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8964","last_scrub_stamp":"2024-09-18T14:32:56.777338+0000","last_deep_scrub":"47'8964","last_deep_scrub_stamp":"2024-09-18T14:32:56.777338+0000","last_clean_scrub_stamp":"2024-09-18T14:32:56.777338+0000","objects_scrubbed":4482,"log_size":8964,"log_dups_size":0,"ondisk_log_size":8964,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T18:26:11.615377+0000","scrub_duration":15.348717383,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":30711,"num_objects":4482,"num_object_clones":0,"num_object_copies":8964,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":4505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4483,"num_bytes_recovered":26232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"47'9273","reported_seq":9518,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:33:27.075507+0000","last_change":"2024-09-18T14:33:27.075430+0000","last_active":"2024-09-18T14:33:27.075507+0000","last_peered":"2024-09-18T14:33:27.075507+0000","last_clean":"2024-09-18T14:33:27.075507+0000","last_became_active":"2024-09-18T14:25:50.057896+0000","last_became_peered":"2024-09-18T14:25:50.057896+0000","last_unstale":"2024-09-18T14:33:27.075507+0000","last_undegraded":"2024-09-18T14:33:27.075507+0000","last_fullsized":"2024-09-18T14:33:27.075507+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9273","last_scrub_stamp":"2024-09-18T14:33:27.075320+0000","last_deep_scrub":"47'9273","last_deep_scrub_stamp":"2024-09-18T14:33:27.075320+0000","last_clean_scrub_stamp":"2024-09-18T14:33:27.075320+0000","objects_scrubbed":4636,"log_size":9273,"log_dups_size":0,"ondisk_log_size":9273,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":14,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T02:06:20.950544+0000","scrub_duration":13.414741224,"objects_trimmed":0,"snaptrim_duration":0.00079432600000000004,"stat_sum":{"num_bytes":30884,"num_objects":4636,"num_object_clones":0,"num_object_copies":9272,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4637,"num_write_kb":4661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4637,"num_bytes_recovered":26270,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"47'8980","reported_seq":9236,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:57.037673+0000","last_change":"2024-09-18T14:32:57.037629+0000","last_active":"2024-09-18T14:32:57.037673+0000","last_peered":"2024-09-18T14:32:57.037673+0000","last_clean":"2024-09-18T14:32:57.037673+0000","last_became_active":"2024-09-18T14:28:37.093965+0000","last_became_peered":"2024-09-18T14:28:37.093965+0000","last_unstale":"2024-09-18T14:32:57.037673+0000","last_undegraded":"2024-09-18T14:32:57.037673+0000","last_fullsized":"2024-09-18T14:32:57.037673+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'8980","last_scrub_stamp":"2024-09-18T14:32:57.037498+0000","last_deep_scrub":"47'8980","last_deep_scrub_stamp":"2024-09-18T14:32:57.037498+0000","last_clean_scrub_stamp":"2024-09-18T14:32:57.037498+0000","objects_scrubbed":4490,"log_size":8980,"log_dups_size":0,"ondisk_log_size":8980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-19T20:56:35.060495+0000","scrub_duration":15.365038688,"objects_trimmed":0,"snaptrim_duration":0.00062271099999999997,"stat_sum":{"num_bytes":17605,"num_objects":4490,"num_object_clones":0,"num_object_copies":8980,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4490,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4490,"num_write_kb":4502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4490,"num_bytes_recovered":13116,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"47'9062","reported_seq":9292,"reported_epoch":47,"state":"active+clean","last_fresh":"2024-09-18T14:32:54.591111+0000","last_change":"2024-09-18T14:32:54.591048+0000","last_active":"2024-09-18T14:32:54.591111+0000","last_peered":"2024-09-18T14:32:54.591111+0000","last_clean":"2024-09-18T14:32:54.591111+0000","last_became_active":"2024-09-18T14:29:35.496135+0000","last_became_peered":"2024-09-18T14:29:35.496135+0000","last_unstale":"2024-09-18T14:32:54.591111+0000","last_undegraded":"2024-09-18T14:32:54.591111+0000","last_fullsized":"2024-09-18T14:32:54.591111+0000","mapping_epoch":40,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"47'9062","last_scrub_stamp":"2024-09-18T14:32:54.590936+0000","last_deep_scrub":"47'9062","last_deep_scrub_stamp":"2024-09-18T14:32:54.590936+0000","last_clean_scrub_stamp":"2024-09-18T14:32:54.590936+0000","objects_scrubbed":4531,"log_size":9062,"log_dups_size":0,"ondisk_log_size":9062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":16,"scrub_schedule":"periodic scrub scheduled @ 2024-09-20T01:46:39.935295+0000","scrub_duration":15.146936148,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56991,"num_objects":4531,"num_object_clones":0,"num_object_copies":9062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4531,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4531,"num_write_kb":4579,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4531,"num_bytes_recovered":52464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":285461,"num_objects":36236,"num_object_clones":0,"num_object_copies":72472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36236,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36236,"num_write_kb":36464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35177,"num_bytes_recovered":249266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":811008,"data_stored":629697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72473,"ondisk_log_size":72473,"up":16,"acting":16,"num_store_stats":4}],"osd_stats":[{"osd":3,"up_from":0,"seq":124554051690,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":0,"seq":107374182507,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":9,"seq":38654705997,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298044,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073796,"statfs":{"total":96636764160,"available":96331567104,"internally_reserved":0,"allocated":708608,"data_stored":394763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Sep 18 14:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.45200000000000001,"15min":0.51000000000000001},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":6.758},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.57399999999999995,"15min":0.59799999999999998},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.24199999999999999},"max":{"1min":0.80000000000000004,"5min":0.99199999999999999,"15min":6.7859999999999996},"last":0.70999999999999996}]},{"osd":2,"last update":"Wed Sep 18 14:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.50600000000000001,"15min":0.48899999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.79300000000000004,"5min":1.357,"15min":1.357},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.55700000000000005,"15min":1.9239999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.26600000000000001},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":46.807000000000002},"last":0.61499999999999999}]},{"osd":3,"last update":"Wed Sep 18 14:33:16 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.53300000000000003,"15min":0.54100000000000004},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.28599999999999998},"max":{"1min":0.72999999999999998,"5min":0.81399999999999995,"15min":1.0309999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.65600000000000003,"15min":0.64100000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.92000000000000004,"5min":0.96399999999999997,"15min":1.1339999999999999},"last":0.82499999999999996}]}]},{"osd":0,"up_from":9,"seq":38654705997,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298044,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":94073796,"statfs":{"total":96636764160,"available":96331567104,"internally_reserved":0,"allocated":708608,"data_stored":394504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Sep 18 14:33:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.46100000000000002,"15min":0.502},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.70399999999999996,"5min":1.3959999999999999,"15min":1.3959999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.55000000000000004,"15min":0.60199999999999998},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26500000000000001},"max":{"1min":0.81899999999999995,"5min":1.542,"15min":1.542},"last":0.80500000000000005}]},{"osd":2,"last update":"Wed Sep 18 14:33:02 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52700000000000002,"15min":0.51200000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.26300000000000001},"max":{"1min":0.80000000000000004,"5min":0.84199999999999997,"15min":0.84999999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.247},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":3.1920000000000002},"last":0.70499999999999996}]},{"osd":3,"last update":"Wed Sep 18 14:33:14 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.61399999999999999,"15min":0.61699999999999999},"min":{"1min":0.318,"5min":0.318,"15min":0.28799999999999998},"max":{"1min":2.895,"5min":2.895,"15min":2.895},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59799999999999998,"15min":0.61099999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.317},"max":{"1min":0.88100000000000001,"5min":1.008,"15min":1.008},"last":0.63600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":319488,"data_stored":249247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":172032,"data_stored":131203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-18T14:33:41.414 DEBUG:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-18T14:33:41.735 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-18T14:33:41.735 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-18T14:33:41.735 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-18T14:33:41.735 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:33:41.816 INFO:tasks.ceph.osd.0:Stopped 2024-09-18T14:33:41.816 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-18T14:33:41.816 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:33:41.896 INFO:tasks.ceph.osd.1:Stopped 2024-09-18T14:33:41.896 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-18T14:33:41.896 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:33:47.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-18T14:33:47.998 INFO:tasks.ceph.osd.2:Stopped 2024-09-18T14:33:47.998 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-18T14:33:47.998 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:33:52.963 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-18T14:33:52.963 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-18T14:33:54.099 INFO:tasks.ceph.osd.3:Stopped 2024-09-18T14:33:54.099 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-18T14:33:54.100 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-18T14:33:54.100 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:33:54.129 INFO:tasks.ceph.mgr.x:Stopped 2024-09-18T14:33:54.129 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-18T14:33:54.129 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-18T14:33:54.130 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:33:54.152 INFO:tasks.ceph.mon.a:Stopped 2024-09-18T14:33:54.152 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-18T14:33:54.152 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:33:54.167 INFO:tasks.ceph.mon.b:Stopped 2024-09-18T14:33:54.167 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-18T14:33:54.167 INFO:teuthology.orchestra.run:waiting for 300 2024-09-18T14:33:54.187 INFO:tasks.ceph.mon.c:Stopped 2024-09-18T14:33:54.187 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-18T14:33:54.188 DEBUG:teuthology.orchestra.run.smithi016:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v '\(MON_DOWN\)' | head -n 1 2024-09-18T14:33:54.265 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:33:54.265 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-18T14:33:54.652 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:33:54.652 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-18T14:33:54.950 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:33:54.950 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-18T14:33:55.105 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:33:55.105 DEBUG:teuthology.orchestra.run.smithi016:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-18T14:33:55.267 INFO:tasks.ceph:Archiving mon data... 2024-09-18T14:33:55.268 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080/data/mon.a.tgz 2024-09-18T14:33:55.268 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-18T14:33:55.290 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.8wrIfWEp6v 2024-09-18T14:33:55.290 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.8wrIfWEp6v 2024-09-18T14:33:55.423 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.8wrIfWEp6v 2024-09-18T14:33:55.475 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.8wrIfWEp6v is 252KB 2024-09-18T14:33:55.541 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.8wrIfWEp6v 2024-09-18T14:33:55.551 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080/data/mon.b.tgz 2024-09-18T14:33:55.551 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-18T14:33:55.558 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.UiesFA5KsZ 2024-09-18T14:33:55.558 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.UiesFA5KsZ 2024-09-18T14:33:55.629 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.UiesFA5KsZ 2024-09-18T14:33:55.651 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.UiesFA5KsZ is 255KB 2024-09-18T14:33:55.671 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.UiesFA5KsZ 2024-09-18T14:33:55.678 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080/data/mon.c.tgz 2024-09-18T14:33:55.678 DEBUG:teuthology.orchestra.run.smithi016:> mktemp 2024-09-18T14:33:55.728 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.84YvJoQRsG 2024-09-18T14:33:55.729 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.84YvJoQRsG 2024-09-18T14:33:55.797 DEBUG:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.84YvJoQRsG 2024-09-18T14:33:55.819 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.84YvJoQRsG is 255KB 2024-09-18T14:33:55.881 DEBUG:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.84YvJoQRsG 2024-09-18T14:33:55.887 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-18T14:33:55.888 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-18T14:33:55.912 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-18T14:33:55.913 DEBUG:teuthology.orchestra.run.smithi016:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-18T14:33:55.940 INFO:teuthology.orchestra.run.smithi016.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-18T14:33:55.946 INFO:tasks.ceph:Archiving crash dumps... 2024-09-18T14:33:55.948 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080/remote/smithi016/crash 2024-09-18T14:33:55.949 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-18T14:33:55.963 INFO:tasks.ceph:Compressing logs... 2024-09-18T14:33:55.963 DEBUG:teuthology.orchestra.run.smithi016:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-18T14:33:56.028 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36383.log 2024-09-18T14:33:56.028 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40102.log 2024-09-18T14:33:56.029 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31783.log 2024-09-18T14:33:56.029 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36383.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36383.log.gz-5 2024-09-18T14:33:56.029 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.47101.log 2024-09-18T14:33:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31783.log: /var/log/ceph/ceph-client.admin.40102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31783.log.gz 2024-09-18T14:33:56.031 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32944.log 2024-09-18T14:33:56.031 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40102.log.gz 2024-09-18T14:33:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32917.log 2024-09-18T14:33:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47101.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47101.log.gz 2024-09-18T14:33:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62656.log 2024-09-18T14:33:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32944.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32944.log.gz 2024-09-18T14:33:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32917.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32917.log.gz 2024-09-18T14:33:56.032 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32485.log 2024-09-18T14:33:56.034 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31135.log 2024-09-18T14:33:56.034 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62656.log.gz 2024-09-18T14:33:56.035 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36639.log 2024-09-18T14:33:56.035 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32485.log.gz 2024-09-18T14:33:56.035 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59278.log 2024-09-18T14:33:56.035 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31135.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31135.log.gz 2024-09-18T14:33:56.035 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36639.log.gz 2024-09-18T14:33:56.035 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35195.log 2024-09-18T14:33:56.037 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66067.log 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59278.log.gz 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52232.log 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35195.log.gz 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34763.log 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66067.log.gz 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62251.log 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52232.log: /var/log/ceph/ceph-client.admin.34763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52232.log.gz 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34763.log.gz-5 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.33052.log 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34142.log 2024-09-18T14:33:56.038 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62251.log.gz 2024-09-18T14:33:56.039 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43157.log 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33052.log: /var/log/ceph/ceph-client.admin.34142.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33052.log.gz 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34142.log.gz 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47290.log 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40237.log 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43157.log.gz 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28730.log 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47290.log.gz 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56335.log 2024-09-18T14:33:56.041 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40237.log.gz 2024-09-18T14:33:56.042 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28460.log 2024-09-18T14:33:56.042 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28730.log: gzip /var/log/ceph/ceph-client.admin.56335.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38017.log 2024-09-18T14:33:56.042 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28730.log.gz 2024-09-18T14:33:56.042 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56335.log.gz 2024-09-18T14:33:56.045 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43697.log 2024-09-18T14:33:56.045 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28460.log.gz 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47587.log 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38017.log.gz 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65581.log 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43697.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43697.log.gz 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58360.log 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47587.log.gz 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40021.log 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65581.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65581.log.gz 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26009.log 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58360.log.gz 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37261.log 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40021.log.gz 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57118.log 2024-09-18T14:33:56.046 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26009.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26009.log.gz 2024-09-18T14:33:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.37261.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37855.log 2024-09-18T14:33:56.047 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37261.log.gz 2024-09-18T14:33:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57118.log.gz 2024-09-18T14:33:56.047 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36801.log 2024-09-18T14:33:56.050 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27866.log 2024-09-18T14:33:56.050 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37855.log.gz 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43373.log 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36801.log.gz 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65689.log 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27866.log.gz 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43886.log 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43373.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43373.log.gz 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65202.log 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65689.log.gz 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63898.log 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43886.log.gz 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60197.log 2024-09-18T14:33:56.051 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51556.log 2024-09-18T14:33:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63898.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65202.log.gz 2024-09-18T14:33:56.052 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63898.log.gz 2024-09-18T14:33:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48910.log 2024-09-18T14:33:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60197.log.gz 2024-09-18T14:33:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33484.log 2024-09-18T14:33:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51556.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51556.log.gz 2024-09-18T14:33:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48910.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48910.log.gz 2024-09-18T14:33:56.052 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60575.log 2024-09-18T14:33:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29758.log 2024-09-18T14:33:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33484.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33484.log.gz 2024-09-18T14:33:56.056 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49774.log 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60575.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60575.log.gz 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56875.log 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29758.log.gz 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39070.log 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49774.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49774.log.gz-5 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.35681.log 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56875.log.gz 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41047.log 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39070.log.gz 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66229.log 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35681.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35681.log.gz 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29081.log 2024-09-18T14:33:56.057 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41047.log.gz 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36113.log 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66229.log.gz 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49612.log 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29081.log.gz 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59008.log 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36113.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36113.log.gz 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65040.log 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49612.log.gz 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34439.log 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59008.log: /var/log/ceph/ceph-client.admin.65040.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59008.log.gz 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65527.log 2024-09-18T14:33:56.058 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65040.log.gz 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50287.log 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34439.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34439.log.gz 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53188.log 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65527.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65527.log.gz 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58981.log 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50287.log.gz 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.53188.log: -- /var/log/ceph/ceph-client.admin.53354.log 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53188.log.gz 2024-09-18T14:33:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58603.log 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35897.log 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53354.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58981.log.gz 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53354.log.gz 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57253.log 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58603.log.gz 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48613.log 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35897.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35897.log.gz 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57253.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57253.log.gz 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53406.log 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55768.log 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48613.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48613.log.gz 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26825.log 2024-09-18T14:33:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53406.log.gz 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31243.log 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55768.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55768.log.gz 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59845.log 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26825.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26825.log.gz 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33619.log 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31243.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31243.log.gz 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59845.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.59845.log.gz-5 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.30109.log 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63790.log 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33619.log.gz 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59170.log 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30109.log.gz 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37071.log 2024-09-18T14:33:56.067 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63790.log.gz 2024-09-18T14:33:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47992.log 2024-09-18T14:33:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59170.log.gz 2024-09-18T14:33:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45453.log 2024-09-18T14:33:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37071.log.gz 2024-09-18T14:33:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47992.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.47992.log.gz--verbose 2024-09-18T14:33:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-mon.a.log 2024-09-18T14:33:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45453.log.gz 2024-09-18T14:33:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41993.log 2024-09-18T14:33:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65500.log 2024-09-18T14:33:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44994.log 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41993.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43913.log 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65500.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41993.log.gz 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65500.log.gz 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64330.log 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44994.log.gz 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63115.log 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43913.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43913.log.gz 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32998.log 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64330.log.gz 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.63115.log: -- /var/log/ceph/ceph-client.admin.61467.log 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63115.log.gz 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47830.log 2024-09-18T14:33:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32998.log.gz 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49531.log 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61467.log.gz 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48775.log 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47830.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47830.log.gz 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59737.log 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49531.log.gz 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61548.log 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48775.log.gz 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48181.log 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59737.log.gz 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61548.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39016.log 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61548.log.gz 2024-09-18T14:33:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27704.log 2024-09-18T14:33:56.076 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48181.log.gz 2024-09-18T14:33:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51745.log 2024-09-18T14:33:56.076 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz 2024-09-18T14:33:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27704.log.gz 2024-09-18T14:33:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35708.log 2024-09-18T14:33:56.081 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51745.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51745.log.gz 2024-09-18T14:33:56.081 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47749.log 2024-09-18T14:33:56.081 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62413.log 2024-09-18T14:33:56.081 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50071.log 2024-09-18T14:33:56.081 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47749.log.gz -- replaced with /var/log/ceph/ceph-client.admin.35708.log.gz 2024-09-18T14:33:56.081 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40048.log 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62413.log.gz 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37234.log 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50071.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50071.log.gz 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45372.log 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40048.log.gz 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48883.log 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37234.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37234.log.gz 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55633.log 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45372.log.gz 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48883.log: /var/log/ceph/ceph-client.admin.37882.log 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48883.log.gz 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34844.log 2024-09-18T14:33:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55633.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55633.log.gz 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63709.log 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37882.log.gz 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28163.log 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34844.log.gz 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36585.log 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63709.log.gz 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49153.log 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28163.log.gz 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61602.log 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36585.log.gz 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55441.log 2024-09-18T14:33:56.083 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49153.log.gz 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45345.log 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61602.log.gz 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.55441.log: -- /var/log/ceph/ceph-client.admin.60548.log 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55441.log.gz 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45345.log.gz 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59926.log 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60548.log.gz 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44940.log 2024-09-18T14:33:56.089 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mon.b.log: /var/log/ceph/ceph-client.admin.59926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56956.log 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59926.log.gz 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49207.log 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44940.log.gz 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56470.log 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56956.log.gz 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43994.log 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49207.log.gz 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25875.log 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56470.log.gz 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55468.log 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43994.log.gz 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59440.log 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25875.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25875.log.gz 2024-09-18T14:33:56.090 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58306.log 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55468.log.gz 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38881.log 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59440.log.gz 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34520.log 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58306.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.58306.log.gz--verbose -- 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66391.log 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35627.log 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38881.log.gz 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34520.log.gz 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36059.log 2024-09-18T14:33:56.091 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66391.log.gz 2024-09-18T14:33:56.101 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-09-18T14:33:56.101 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61710.log 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36059.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35627.log.gz 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36059.log.gz 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63439.log 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35438.log 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61710.log.gz 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57091.log 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63439.log.gz 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62737.log 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35438.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64222.log 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35438.log.gz 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36032.log 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57091.log.gz 2024-09-18T14:33:56.102 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62737.log.gz 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51232.log 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64222.log.gz 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56659.log 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36032.log: /var/log/ceph/ceph-client.admin.51232.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36032.log.gz 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41912.log 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51232.log.gz 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42182.log 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56659.log.gz 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56821.log 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41912.log.gz 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42182.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46885.log 2024-09-18T14:33:56.103 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42182.log.gz 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55522.log 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56821.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.56821.log.gz--verbose 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.27731.log 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46885.log.gz 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43292.log 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55522.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55522.log.gz 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27731.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30433.log 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27731.log.gz 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36612.log 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43292.log.gz 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65743.log 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30433.log.gz 2024-09-18T14:33:56.104 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60332.log 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36612.log.gz 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64114.log 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65743.log.gz 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.60332.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38098.log 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60332.log.gz 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64114.log.gz 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32053.log 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41426.log 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38098.log.gz 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38530.log 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32053.log.gz 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56227.log 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41426.log.gz 2024-09-18T14:33:56.105 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44480.log 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38530.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38530.log.gz 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56632.log 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56227.log.gz 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34250.log 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44480.log.gz 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56632.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56632.log.gz 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29324.log 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34250.log.gz 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59656.log 2024-09-18T14:33:56.106 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29324.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29324.log.gz 2024-09-18T14:33:56.120 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62791.log 2024-09-18T14:33:56.120 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59656.log.gz 2024-09-18T14:33:56.120 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30595.log 2024-09-18T14:33:56.120 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37153.log 2024-09-18T14:33:56.120 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62791.log.gz 2024-09-18T14:33:56.120 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64303.log 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30595.log.gz 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26307.log 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37153.log: /var/log/ceph/ceph-client.admin.64303.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37153.log.gz 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46750.log 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64303.log.gz 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26131.log 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26307.log.gz 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46750.log.gz 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50503.log 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26131.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26131.log.gz 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33764.log 2024-09-18T14:33:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50503.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50503.log.gz 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28541.log 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56578.log 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33764.log.gz 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50017.log 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28541.log.gz 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65608.log 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56578.log.gz 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33376.log 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50017.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50017.log.gz 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39286.log 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65608.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65608.log.gz 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33376.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33376.log.gz-5 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.40291.log 2024-09-18T14:33:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39286.log.gz 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31648.log 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40291.log.gz 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36464.log 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39394.log 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31648.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31648.log.gz 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63007.log 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36464.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36464.log.gz 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30406.log 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39394.log.gz 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30622.log 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63007.log.gz 2024-09-18T14:33:56.123 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34196.log 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30406.log.gz 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48127.log 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30622.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30622.log.gz 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51070.log 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34196.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34196.log.gz 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59116.log 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48127.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48127.log.gz 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35600.log 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51070.log.gz 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57739.log 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59116.log.gz 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36302.log 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35600.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35600.log.gz 2024-09-18T14:33:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50827.log 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57739.log.gz 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38233.log 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36302.log.gz 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27272.log /var/log/ceph/ceph-client.admin.50827.log: 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50827.log.gz 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29054.log 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38233.log.gz 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29243.log 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27272.log.gz 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61494.log 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29054.log.gz 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59494.log /var/log/ceph/ceph-client.admin.29243.log: 2024-09-18T14:33:56.125 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29243.log.gz 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34331.log 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61494.log.gz 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30514.log 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59494.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59494.log.gz 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58144.log 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34331.log.gz 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51772.log 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30514.log.gz 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66877.log 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58144.log.gz 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31432.log 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51772.log.gz 2024-09-18T14:33:56.126 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22174.log 2024-09-18T14:33:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66877.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66877.log.gz 2024-09-18T14:33:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62494.log 2024-09-18T14:33:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31432.log.gz 2024-09-18T14:33:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49909.log 2024-09-18T14:33:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.22174.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22174.log.gz 2024-09-18T14:33:56.153 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51826.log 2024-09-18T14:33:56.153 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62494.log.gz 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51394.log 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49909.log.gz 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59548.log 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51826.log.gz 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30892.log 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51394.log: /var/log/ceph/ceph-client.admin.59548.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.51394.log.gz--verbose 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.37639.log 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59548.log.gz 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32215.log 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30892.log.gz 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61278.log 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37639.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37639.log.gz 2024-09-18T14:33:56.154 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28298.log 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32215.log.gz 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58738.log 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61278.log.gz 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28298.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28298.log.gz 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50746.log 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33025.log 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58738.log.gz 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37720.log 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50746.log.gz 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33025.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33025.log.gz 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50719.log 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37720.log.gz 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47074.log 2024-09-18T14:33:56.155 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42806.log 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57388.log 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50719.log.gz 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47074.log.gz 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41534.log 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42806.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42806.log.gz 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57685.log 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57388.log.gz 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59683.log 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41534.log.gz 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40318.log 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57685.log.gz 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59683.log.gz 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50800.log 2024-09-18T14:33:56.156 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41750.log 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40318.log.gz/var/log/ceph/ceph-client.admin.50800.log: 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.50800.log.gz-5 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.32431.log 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41750.log.gz 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43940.log 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38719.log 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32431.log.gz 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37017.log 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43940.log: /var/log/ceph/ceph-client.admin.38719.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43940.log.gz 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38719.log.gz 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27326.log 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38827.log 2024-09-18T14:33:56.157 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40615.log 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37017.log.gz 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27326.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27326.log.gz 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27542.log 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38827.log.gz 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40615.log.gz 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45048.log 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27542.log.gz 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56092.log 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45048.log.gz 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26611.log 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56092.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56092.log.gz 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45102.log 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26611.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41075.log 2024-09-18T14:33:56.158 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26611.log.gz 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32836.log 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45102.log.gz 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35303.log 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41075.log.gz 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64060.log 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32836.log.gz 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35303.log: /var/log/ceph/ceph-client.admin.66769.log 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35303.log.gz 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48208.log 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64060.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64060.log.gz 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66769.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66769.log.gz 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34115.log 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46723.log 2024-09-18T14:33:56.159 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48208.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48208.log.gz 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34115.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34115.log.gz 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62602.log 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39751.log 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46723.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46723.log.gz 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39451.log 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62602.log.gz 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57010.log 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39751.log: /var/log/ceph/ceph-client.admin.39451.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39751.log.gz-5 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.30136.log 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39451.log.gz 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36855.log 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57010.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57010.log.gz 2024-09-18T14:33:56.160 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31702.log 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30136.log.gz 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57064.log 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36855.log.gz 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31702.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31702.log.gz 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58009.log 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56362.log 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57064.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57064.log.gz 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29189.log 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58009.log.gz 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47479.log 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56362.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.56362.log.gz-5 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.51043.log 2024-09-18T14:33:56.161 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29189.log.gz 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39232.log 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47479.log.gz 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42563.log 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51043.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51043.log.gz 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56848.log /var/log/ceph/ceph-client.admin.39232.log: 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39232.log.gz 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57577.log 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42563.log.gz 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26604.log 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56848.log.gz 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48829.log 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57577.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57577.log.gz 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26604.log.gz 2024-09-18T14:33:56.162 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67039.log 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34952.log 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48829.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48829.log.gz 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59359.log 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.67039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28433.log 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67039.log.gz 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34952.log: /var/log/ceph/ceph-client.admin.59359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34952.log.gz 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59359.log.gz 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55876.log 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39640.log 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28433.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28433.log.gz 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55957.log 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55876.log: /var/log/ceph/ceph-client.admin.39640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55876.log.gz 2024-09-18T14:33:56.163 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39640.log.gz 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33845.log 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62899.log 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55957.log.gz 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38071.log 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33845.log: /var/log/ceph/ceph-client.admin.62899.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33845.log.gz 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30460.log 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62899.log.gz 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63736.log 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38071.log.gz 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65878.log 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30460.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30460.log.gz 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59332.log 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63736.log.gz 2024-09-18T14:33:56.164 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50665.log 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65878.log.gz 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43319.log 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59332.log.gz 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50665.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30190.log 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50665.log.gz 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40561.log 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43319.log: /var/log/ceph/ceph-client.admin.30190.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43319.log.gz 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63250.log 2024-09-18T14:33:56.165 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30190.log.gz 2024-09-18T14:33:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55387.log 2024-09-18T14:33:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40561.log.gz 2024-09-18T14:33:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48667.log 2024-09-18T14:33:56.217 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63250.log.gz 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27033.log 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55387.log.gz 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37044.log 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48667.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48667.log.gz 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.27033.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44264.log 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27033.log.gz 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64634.log 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37044.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37044.log.gz 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45669.log 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44264.log: /var/log/ceph/ceph-client.admin.64634.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44264.log.gz 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34790.log 2024-09-18T14:33:56.218 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64634.log.gz 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45750.log 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45669.log.gz 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.34790.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29162.log 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34790.log.gz 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37558.log 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45750.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45750.log.gz 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.29162.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47236.log 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29162.log.gz 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37558.log.gz 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57307.log 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47236.log.gz 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66985.log 2024-09-18T14:33:56.219 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57307.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57307.log.gz 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65905.log 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66985.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45939.log 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66985.log.gz 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55984.log 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65905.log.gz 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41507.log 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29542.log 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55984.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55984.log.gz 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50260.log 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41507.log.gz 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57199.log 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29542.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29542.log.gz 2024-09-18T14:33:56.220 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50260.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50260.log.gz 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46858.log 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57199.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57199.log.gz 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27380.log 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51610.log 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25775.log 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27380.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27380.log.gz 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46858.log.gz 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64168.log 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51610.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51610.log.gz 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29704.log 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25775.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25775.log.gz 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64168.log.gz 2024-09-18T14:33:56.221 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40156.log 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29704.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29704.log.gz 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30757.log 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40156.log.gz 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58252.log 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30757.log.gz 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52286.log 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43076.log 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39205.log 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52286.log.gz 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58252.log.gz 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40426.log 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43076.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43076.log.gz 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42968.log 2024-09-18T14:33:56.222 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39205.log.gz 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66742.log 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40426.log.gz 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64661.log 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42968.log.gz 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60711.log 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66742.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66742.log.gz 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43508.log 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64661.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64661.log.gz 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50692.log 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60711.log.gz 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49558.log 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43508.log.gz 2024-09-18T14:33:56.223 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65067.log 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50692.log: /var/log/ceph/ceph-client.admin.49558.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49558.log.gz 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50692.log.gz 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31405.log 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65067.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.65067.log.gz--verbose 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.25848.log 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30568.log 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31405.log.gz 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32134.log 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25848.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25848.log.gz 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45183.log 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30568.log: /var/log/ceph/ceph-client.admin.32134.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50611.log 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32134.log.gz 2024-09-18T14:33:56.224 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30568.log.gz 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45183.log.gz 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37585.log 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61737.log 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50611.log: /var/log/ceph/ceph-client.admin.37585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37585.log.gz 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50611.log.gz 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38368.log 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61737.log.gz 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42779.log 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38368.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38368.log.gz 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39259.log 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42779.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42779.log.gz 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55360.log 2024-09-18T14:33:56.225 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40966.log 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39259.log.gz 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53161.log 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55360.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55360.log.gz 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59062.log 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40966.log.gz 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61683.log 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53161.log.gz 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59062.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31891.log 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59062.log.gz 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61683.log.gz 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51961.log 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31891.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31891.log.gz 2024-09-18T14:33:56.226 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42209.log 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51961.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51961.log.gz 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47452.log 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51340.log 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42209.log.gz 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32566.log 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47452.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47452.log.gz 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51340.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51340.log.gz 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66175.log 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62926.log 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32566.log.gz 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66175.log.gzgzip 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.36410.log 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60900.log 2024-09-18T14:33:56.227 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62926.log.gz 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37531.log 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36410.log.gz 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60900.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60900.log.gz-5 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.36936.log 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62521.log 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37531.log: /var/log/ceph/ceph-client.admin.36936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37531.log.gz 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36936.log.gz 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.58684.log 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62521.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62521.log.gz 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30055.log 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30973.log 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58684.log: /var/log/ceph/ceph-client.admin.30055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45480.log 2024-09-18T14:33:56.228 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30055.log.gz 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58684.log.gz 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50152.log 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30973.log: /var/log/ceph/ceph-client.admin.45480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49261.log 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30973.log.gz 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45480.log.gz 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.50152.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.22114.log 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50152.log.gz 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30487.log 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49261.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49261.log.gz 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.22114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57982.log 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22114.log.gz 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34736.log 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30487.log: /var/log/ceph/ceph-client.admin.57982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51475.log 2024-09-18T14:33:56.229 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30487.log.gz 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57982.log.gz 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35357.log 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34736.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34736.log.gz 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52015.log 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51475.log: /var/log/ceph/ceph-client.admin.35357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58468.log 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51475.log.gz 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35357.log.gz 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67067.log 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52015.log.gz 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37396.log 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58468.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58522.log 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.67067.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58468.log.gz 2024-09-18T14:33:56.230 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67067.log.gz 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37396.log.gz 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60602.log 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58522.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66148.log 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58522.log.gz 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66661.log 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60602.log: 27.8% -- replaced with /var/log/ceph/ceph-client.admin.45939.log.gz 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60602.log.gz 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37747.log 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66148.log.gz 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48721.log 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60873.log 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%/var/log/ceph/ceph-client.admin.37747.log: -- replaced with /var/log/ceph/ceph-client.admin.66661.log.gz 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37747.log.gz 2024-09-18T14:33:56.231 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48721.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64580.log 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48721.log.gz 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65229.log 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60873.log.gz 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46209.log 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64580.log.gz 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50449.log 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65229.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65229.log.gz 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36140.log 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46209.log.gz 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42263.log 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50449.log.gz 2024-09-18T14:33:56.232 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50476.log 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36140.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36140.log.gz 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42617.log 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42263.log.gz 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58198.log 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66094.log 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50476.log.gz 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44588.log 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58198.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42617.log.gz 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58198.log.gz 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35276.log 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66094.log.gz 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60170.log 2024-09-18T14:33:56.233 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44588.log.gz 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50422.log 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35276.log: /var/log/ceph/ceph-client.admin.60170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35276.log.gz 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29650.log 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60170.log.gz 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50422.log: /var/log/ceph/ceph-client.admin.35573.log 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50422.log.gz 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38746.log 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29650.log: /var/log/ceph/ceph-client.admin.35573.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35573.log.gz 2024-09-18T14:33:56.234 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29650.log.gz 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59575.log 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50395.log /var/log/ceph/ceph-client.admin.38746.log: 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38746.log.gz 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45507.log 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59575.log.gz 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48235.log 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66526.log 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45507.log.gz 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50395.log.gz 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35951.log 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48235.log: /var/log/ceph/ceph-client.admin.66526.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48235.log.gz 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31216.log 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66526.log.gz 2024-09-18T14:33:56.235 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35951.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35951.log.gz 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49963.log 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31216.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31216.log.gz 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.40264.log 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52646.log 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49963.log.gz 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34655.log 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40264.log: /var/log/ceph/ceph-client.admin.52646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56497.log 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40264.log.gz 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52646.log.gz 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44183.log 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34655.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34655.log.gz 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52177.log 2024-09-18T14:33:56.236 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56497.log.gz 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29488.log 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44183.log.gz 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.52177.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.66607.log 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52177.log.gz 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52624.log 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29488.log.gz 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64796.log 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66607.log.gz 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52624.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59386.log 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52624.log.gz 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37936.log 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64796.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64796.log.gz 2024-09-18T14:33:56.237 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28568.log 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59386.log.gz 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57847.log 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37936.log.gz 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53593.log 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28568.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28568.log.gz 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30379.log 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57847.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57847.log.gz 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51313.log 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53593.log.gz 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27137.log 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30379.log.gz 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60684.log 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45534.log 2024-09-18T14:33:56.238 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51313.log.gz 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27137.log: /var/log/ceph/ceph-client.admin.60684.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27137.log.gz -- replaced with /var/log/ceph/ceph-client.admin.60684.log.gz 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26280.log 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44534.log 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45534.log: /var/log/ceph/ceph-client.admin.26280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45534.log.gz 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28973.log 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26280.log.gz 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52956.log 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44534.log.gz 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43265.log 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28973.log.gz 2024-09-18T14:33:56.239 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28001.log 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59899.log 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52956.log.gz 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38287.log 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28001.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43265.log.gz 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28001.log.gz 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.59899.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55795.log 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59899.log.gz 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31621.log 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38287.log: gzip /var/log/ceph/ceph-client.admin.55795.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38665.log 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38287.log.gz 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55795.log.gz 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38206.log 2024-09-18T14:33:56.240 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62089.log 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31621.log.gz 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38665.log: /var/log/ceph/ceph-client.admin.38206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38206.log.gz 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38665.log.gz 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42101.log 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58549.log 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62089.log: gzip /var/log/ceph/ceph-client.admin.42101.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.33214.log 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42101.log.gz 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62089.log.gz 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35870.log 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58549.log: /var/log/ceph/ceph-client.admin.33214.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58549.log.gz 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33214.log.gz 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48397.log 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.35870.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.46696.log 2024-09-18T14:33:56.241 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35870.log.gz 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66040.log 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48397.log: /var/log/ceph/ceph-client.admin.46696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55495.log 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46696.log.gz 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48397.log.gz 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56308.log 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66040.log: /var/log/ceph/ceph-client.admin.55495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31837.log 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66040.log.gz 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55495.log.gz 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60954.log 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56308.log.gz 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38557.log 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50935.log 2024-09-18T14:33:56.242 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31837.log.gz 2024-09-18T14:33:56.243 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60954.log.gz 2024-09-18T14:33:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60143.log 2024-09-18T14:33:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38557.log.gz 2024-09-18T14:33:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50935.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30028.log 2024-09-18T14:33:56.243 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50935.log.gz 2024-09-18T14:33:56.243 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60143.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60143.log.gz 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30811.log 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30244.log 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30028.log.gz 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39340.log 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62278.log 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30811.log.gz 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30244.log.gz 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28352.log 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39340.log.gz 2024-09-18T14:33:56.299 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52866.log 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62278.log.gz 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63925.log 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28352.log.gz 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28865.log 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52866.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52866.log.gz 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46453.log 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63925.log.gz 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.28865.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43724.log 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28865.log.gz 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55414.log 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46453.log.gz 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56551.log 2024-09-18T14:33:56.300 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43724.log.gz 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28109.log 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55414.log.gz 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50233.log 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56551.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56551.log.gz 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44237.log 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28109.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28109.log.gz 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53566.log 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50233.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50233.log.gz 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39586.log 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44237.log.gz 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43967.log 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53566.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53566.log.gz 2024-09-18T14:33:56.301 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26508.log 0.0% 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39586.log.gz 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38800.log 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43967.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43967.log.gz 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60278.log 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26508.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26508.log.gz 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36828.log 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38800.log.gz 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60278.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60278.log.gz 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62872.log 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34493.log 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36828.log.gz 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62872.log.gz 2024-09-18T14:33:56.302 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31729.log 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34493.log.gz 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63952.log 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27839.log 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31729.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31729.log.gz 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56173.log 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63952.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63952.log.gz 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51799.log 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27839.log.gz 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43211.log 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56173.log.gz 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43832.log 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51799.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51799.log.gz 2024-09-18T14:33:56.303 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36248.log 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43211.log.gz 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.43832.log: -- /var/log/ceph/ceph-client.admin.26399.log 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43832.log.gz 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47020.log 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36248.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36248.log.gz 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57415.log 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26399.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26399.log.gz 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36963.log 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47020.log.gz 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57415.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57415.log.gz 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26822.log 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66310.log 2024-09-18T14:33:56.304 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36963.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36963.log.gz 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26822.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45210.log 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26822.log.gz 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63844.log 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66310.log.gz 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66418.log 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45210.log.gz 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63844.log.gz 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43022.log 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66121.log 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66418.log.gz 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35924.log 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43022.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43022.log.gz 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60629.log 2024-09-18T14:33:56.305 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66121.log.gz 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45993.log 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35924.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35924.log.gz 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43184.log 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60629.log.gz 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45993.log.gz 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64472.log 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46290.log 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43184.log.gz 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49693.log 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64472.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64472.log.gz 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46290.log.gz 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28703.log 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49693.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.49693.log.gz--verbose 2024-09-18T14:33:56.306 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.52150.log 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34628.log 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28703.log.gz 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52150.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43859.log 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52150.log.gz 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58657.log 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34628.log.gz 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43346.log 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43859.log.gz 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37099.log 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58657.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58657.log.gz 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43346.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43346.log.gz 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58414.log 2024-09-18T14:33:56.307 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49747.log 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37099.log.gz 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58414.log.gz 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63574.log 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49747.log: /var/log/ceph/ceph-client.admin.41183.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49747.log.gz 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42941.log 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61386.log 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63574.log.gz 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42941.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42941.log.gz 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46020.log 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61386.log.gz 2024-09-18T14:33:56.308 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59872.log 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46020.log.gz 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41561.log 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59872.log.gz 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66445.log 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41561.log: 0.0%/var/log/ceph/ceph-client.admin.66445.log: -- replaced with /var/log/ceph/ceph-client.admin.41561.log.gz 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66445.log.gz 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26391.log 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28244.log 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56983.log 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26391.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26391.log.gz 2024-09-18T14:33:56.309 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27245.log 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28244.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28244.log.gz 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56983.log: /var/log/ceph/ceph-client.admin.65473.log 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56983.log.gz 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34169.log 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27245.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27245.log.gz 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65797.log 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65473.log.gz 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49639.log 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34169.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34169.log.gz 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65797.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65797.log.gz 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42887.log 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26015.log 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49639.log.gz 2024-09-18T14:33:56.310 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28406.log 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42887.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42887.log.gz 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63142.log 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26015.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26015.log.gz 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42047.log 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28406.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28406.log.gz 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52580.log 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63142.log.gz 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61791.log 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52580.log.gz 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25909.log 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56605.log 2024-09-18T14:33:56.311 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61791.log.gz 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39421.log 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25909.log.gz 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60413.log 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29405.log 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56605.log.gz 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39421.log.gz 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60413.log.gz 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27650.log 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28190.log 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29405.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29405.log.gz 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65148.log 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27650.log.gz 2024-09-18T14:33:56.312 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28190.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27218.log 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28190.log.gz 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65148.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65148.log.gz 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44669.log 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35546.log 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27218.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27218.log.gz 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65094.log 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44669.log.gz 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61764.log 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35546.log: 0.0%/var/log/ceph/ceph-client.admin.65094.log: -- replaced with /var/log/ceph/ceph-client.admin.35546.log.gz 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26497.log 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65094.log.gz 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57334.log 2024-09-18T14:33:56.313 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61764.log.gz 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26497.log.gz 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53513.log 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46615.log 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57334.log.gz 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27947.log 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53513.log.gz 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56065.log 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46615.log.gz 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59980.log 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27947.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27947.log.gz 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56065.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.56065.log.gz-5 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.47668.log 2024-09-18T14:33:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45642.log 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59980.log.gz 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47668.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47668.log.gz 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63601.log 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27191.log 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45642.log.gz 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60386.log 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63601.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63601.log.gz 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52918.log 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27191.log.gz 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30001.log 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60386.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60386.log.gz 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26253.log 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52918.log.gz 2024-09-18T14:33:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33872.log 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39367.log 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30001.log.gz 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36329.log 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26253.log: /var/log/ceph/ceph-client.admin.33872.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33872.log.gz 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.26253.log.gz-5 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.55576.log 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39367.log.gz 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36329.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36329.log.gz 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60034.log 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55576.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55576.log.gz 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61143.log 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31459.log 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60034.log: 28.6% -- replaced with /var/log/ceph/ceph-client.admin.60034.log.gz 2024-09-18T14:33:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61143.log.gz 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31459.log.gz 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52734.log 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33187.log 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39967.log 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52734.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52734.log.gz 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52690.log 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33187.log.gz 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39967.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29812.log 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39967.log.gz 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48073.log 2024-09-18T14:33:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52690.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52690.log.gz 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28757.log 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29812.log.gz 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48073.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48073.log.gz 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61197.log 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28757.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28757.log.gz 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65256.log 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39151.log 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53408.log 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61197.log.gz 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65256.log.gz 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26612.log 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39151.log.gz 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.53408.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48451.log 2024-09-18T14:33:56.318 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53408.log.gz 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40183.log /var/log/ceph/ceph-client.admin.26612.log: 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26612.log.gz 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41804.log 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48451.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48451.log.gz 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58117.log 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40183.log.gz 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41804.log.gz 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43589.log 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67012.log 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58117.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58117.log.gz 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65419.log 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43589.log.gz 2024-09-18T14:33:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36882.log 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.67012.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67012.log.gz 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46642.log 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65419.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65419.log.gz 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41615.log 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36882.log: /var/log/ceph/ceph-client.admin.46642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36882.log.gz 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46642.log.gz 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38935.log 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41615.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41615.log.gz 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27623.log 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38935.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38935.log.gz 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56794.log 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27623.log.gz 2024-09-18T14:33:56.320 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29839.log 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56794.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56794.log.gz 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26502.log 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22200.log 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29839.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29839.log.gz 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66202.log 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26502.log.gz 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.22200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52367.log 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22200.log.gz 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42752.log 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66202.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66202.log.gz 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.52367.log: -- /var/log/ceph/ceph-client.admin.28919.log 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52367.log.gz 2024-09-18T14:33:56.321 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42752.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42752.log.gz 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27083.log 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28919.log.gz 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49018.log 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51286.log 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27083.log: /var/log/ceph/ceph-client.admin.49018.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.35519.log 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.27083.log.gz 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49018.log.gz 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32080.log 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51286.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51286.log.gz 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35519.log.gz 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44426.log 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32080.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.32080.log.gz 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.49477.log 2024-09-18T14:33:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49234.log 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44426.log: /var/log/ceph/ceph-client.admin.49477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44426.log.gz 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49477.log.gz 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51016.log 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49234.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.49234.log.gz 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.51097.log 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61873.log 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51016.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51016.log.gz 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31270.log 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51097.log.gz 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.61873.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.29027.log 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61873.log.gz 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45264.log 2024-09-18T14:33:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31270.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31270.log.gz 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36909.log 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29027.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29027.log.gz 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52470.log 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45264.log: /var/log/ceph/ceph-client.admin.36909.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36909.log.gz 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45264.log.gz 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32593.log 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40858.log 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52470.log.gz 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64391.log 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32593.log.gz 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28676.log 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40858.log: /var/log/ceph/ceph-client.admin.64391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58225.log 0.0% 2024-09-18T14:33:56.324 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40858.log.gz 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64391.log.gz 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27569.log 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28676.log.gz 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45021.log 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58225.log.gz 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46939.log 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27569.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27569.log.gz 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27488.log 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44642.log 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45021.log.gz 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46939.log.gz 2024-09-18T14:33:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53058.log 2024-09-18T14:33:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32755.log 2024-09-18T14:33:56.371 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27488.log.gz 2024-09-18T14:33:56.371 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44642.log: 0.0%/var/log/ceph/ceph-client.admin.53058.log: -- replaced with /var/log/ceph/ceph-client.admin.44642.log.gz 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50044.log 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53058.log.gz 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64553.log 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32755.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.32755.log.gz--verbose -- 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48046.log 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27920.log 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50044.log.gz 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64553.log.gz 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27893.log 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48046.log.gz 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64878.log 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27920.log.gz 2024-09-18T14:33:56.372 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44210.log 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27893.log.gz 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64878.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64878.log.gz 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45804.log 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57766.log 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44210.log.gz 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43562.log 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30163.log 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57766.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57766.log.gz 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62710.log 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45804.log: /var/log/ceph/ceph-client.admin.43562.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45804.log.gz 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43562.log.gz 2024-09-18T14:33:56.373 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46371.log 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30163.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30163.log.gz 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62710.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65121.log 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62710.log.gz 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29351.log 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46371.log.gz 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38341.log 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65121.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65121.log.gz 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58171.log 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29351.log.gz 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38341.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38341.log.gz 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66688.log 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64607.log 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55849.log 2024-09-18T14:33:56.374 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58171.log.gz 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66688.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66688.log.gz 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26786.log 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64607.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64607.log.gz 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58063.log 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55849.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55849.log.gz 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66472.log 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26786.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26786.log.gz 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46236.log 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26976.log 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58063.log.gz 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47722.log 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46236.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66472.log.gz 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46236.log.gz 2024-09-18T14:33:56.375 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26976.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26976.log.gz 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45696.log 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34304.log 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47722.log.gz 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26712.log 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45696.log.gz 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44075.log 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34304.log.gz 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32188.log 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26712.log.gz 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44075.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44075.log.gz 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44724.log 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58090.log 2024-09-18T14:33:56.376 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32188.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32188.log.gz 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28325.log 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44724.log.gz 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43130.log 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58090.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58090.log.gz 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34385.log 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28325.log.gz 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40129.log 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43130.log.gz 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52891.log 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34385.log.gz 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32404.log 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40129.log.gz 2024-09-18T14:33:56.377 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28784.log 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52891.log.gz 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55687.log 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32404.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32404.log.gz 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52536.log 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28784.log.gz 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55687.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37693.log 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55687.log.gz 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33899.log 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52536.log.gz 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42671.log 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37693.log: /var/log/ceph/ceph-client.admin.33899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30082.log 0.0% 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37693.log.gz 2024-09-18T14:33:56.378 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33899.log.gz 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48154.log 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42671.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42671.log.gz 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41453.log 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30082.log.gz 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48154.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48154.log.gz 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26400.log 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41453.log.gz 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63493.log 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62980.log 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26400.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26400.log.gz 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50881.log 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63493.log: /var/log/ceph/ceph-client.admin.62980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63493.log.gz 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62980.log.gz 2024-09-18T14:33:56.379 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36167.log 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26025.log 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50881.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50881.log.gz 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36167.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36167.log.gz 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49045.log 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 /var/log/ceph/ceph-client.admin.26025.log: --verbose -- /var/log/ceph/ceph-client.admin.44913.log 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26025.log.gz 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60846.log 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49045.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49045.log.gz 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57901.log 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44913.log.gz 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46182.log 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60846.log.gz 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35465.log 2024-09-18T14:33:56.380 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57901.log.gz 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44832.log 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46182.log.gz 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59035.log 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35465.log.gz 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.44832.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.51529.log 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44832.log.gz 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65283.log 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59035.log: 28.1% -- replaced with /var/log/ceph/ceph-client.admin.59035.log.gz 2024-09-18T14:33:56.381 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59602.log 2024-09-18T14:33:56.384 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51529.log.gz 2024-09-18T14:33:56.384 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.65283.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.64851.log 2024-09-18T14:33:56.384 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65283.log.gz 2024-09-18T14:33:56.384 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52259.log 2024-09-18T14:33:56.384 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59602.log.gz 2024-09-18T14:33:56.384 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33349.log 2024-09-18T14:33:56.384 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64851.log.gz 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58576.log 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41318.log 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33349.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52259.log.gz 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33349.log.gz 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27677.log 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58576.log.gz 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57145.log 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41318.log.gz 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65986.log 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27677.log.gz 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57145.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57145.log.gz 2024-09-18T14:33:56.385 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31594.log 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49882.log 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65986.log.gz 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.31594.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32296.log 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31594.log.gz 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44751.log 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49882.log: /var/log/ceph/ceph-client.admin.32296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38260.log 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49882.log.gz 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32296.log.gz 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56281.log 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44751.log.gz 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65338.log 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38260.log: /var/log/ceph/ceph-client.admin.56281.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38260.log.gzgzip 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.46263.log 2024-09-18T14:33:56.386 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56281.log.gz 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65338.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.65338.log.gz 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.60007.log 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53461.log 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46263.log.gz 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61224.log 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.53461.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39043.log 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53461.log.gz 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60007.log.gz 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61224.log.gz 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50341.log 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34682.log 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39043.log: /var/log/ceph/ceph-client.admin.50341.log: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50341.log.gz -- replaced with /var/log/ceph/ceph-client.admin.39043.log.gz 2024-09-18T14:33:56.387 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39505.log 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34682.log.gz 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45858.log 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39505.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39505.log.gz 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29378.log 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45858.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45858.log.gz 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29459.log 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49720.log 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52822.log 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29378.log.gz 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29459.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29459.log.gz 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49720.log.gz 2024-09-18T14:33:56.388 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.17175.log 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62440.log 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52822.log: gzip /var/log/ceph/ceph.tmp-client.admin.17175.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.27029.log 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52822.log.gz 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.17175.log.gz 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58387.log 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62440.log.gz 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27029.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27029.log.gz 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52042.log 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50584.log 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58387.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58387.log.gz 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33295.log 2024-09-18T14:33:56.389 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52042.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52042.log.gz 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49855.log 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50584.log.gz 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59467.log 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33295.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33295.log.gz 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49855.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49855.log.gz 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31027.log 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41858.log 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59467.log.gz 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62683.log 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31027.log.gz 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41858.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66850.log 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41858.log.gz 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51448.log 2024-09-18T14:33:56.390 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62683.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62683.log.gz 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66850.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38989.log 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66850.log.gz 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30217.log 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51448.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51448.log.gz 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63034.log 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38989.log.gz 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64769.log 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30217.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30217.log.gz 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65392.log 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63034.log.gz 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64769.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.64769.log.gz 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.31054.log 2024-09-18T14:33:56.391 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34466.log 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65392.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65392.log.gz 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61954.log 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31054.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31054.log.gz 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40534.log 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34466.log.gz 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46912.log 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61954.log: gzip 0.0%-5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.61954.log.gz 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33079.log 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40534.log.gz 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44021.log 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46912.log.gz 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37288.log 2024-09-18T14:33:56.392 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33079.log.gz 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56011.log 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44021.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44021.log.gz 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48532.log 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37288.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37288.log.gz 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59089.log 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56011.log: /var/log/ceph/ceph-client.admin.48532.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56011.log.gz 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.48532.log.gz 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.37909.log 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.59089.log: -- /var/log/ceph/ceph-client.admin.32458.log 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59089.log.gz 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42860.log 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37909.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.37909.log.gz-5 --verbose 2024-09-18T14:33:56.393 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.36437.log 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32458.log.gz 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34034.log 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42860.log: /var/log/ceph/ceph-client.admin.36437.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42860.log.gz 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.36437.log.gz 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.49288.log 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34034.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34034.log.gz 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60088.log 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49072.log 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49288.log.gz 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60359.log 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60088.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60088.log.gz 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61629.log 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29866.log 2024-09-18T14:33:56.394 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49072.log.gz 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61629.log: gzip -5 --verbose /var/log/ceph/ceph-client.admin.60359.log: -- /var/log/ceph/ceph-client.admin.65770.log 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61629.log.gz 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60359.log.gz 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52492.log 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29866.log.gz 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56524.log 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42698.log 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65770.log.gz 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52492.log: 0.0%/var/log/ceph/ceph-client.admin.56524.log: -- replaced with /var/log/ceph/ceph-client.admin.52492.log.gz 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62359.log 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56524.log.gz 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61656.log 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42698.log: /var/log/ceph/ceph-client.admin.62359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42698.log.gz 2024-09-18T14:33:56.395 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47398.log 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62359.log.gz 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26921.log 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61656.log.gz 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65446.log 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47398.log: /var/log/ceph/ceph-client.admin.26921.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41156.log 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47398.log.gz 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26921.log.gz 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65446.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65446.log.gz 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51178.log 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41156.log.gz 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27974.log 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51178.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51178.log.gz 2024-09-18T14:33:56.396 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28514.log 2024-09-18T14:33:56.397 INFO:teuthology.orchestra.run.smithi016.stderr: 96.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-18T14:33:56.397 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32701.log 2024-09-18T14:33:56.397 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27974.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27974.log.gz 2024-09-18T14:33:56.397 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28514.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28514.log.gz 2024-09-18T14:33:56.397 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29297.log 2024-09-18T14:33:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57469.log 2024-09-18T14:33:56.429 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32701.log: /var/log/ceph/ceph-client.admin.29297.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32701.log.gz 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29297.log.gz 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36086.log 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57469.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57469.log.gz 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31540.log 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36086.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36086.log.gz 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41480.log 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31540.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31540.log.gz 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25905.log 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41480.log: /var/log/ceph/ceph-client.admin.38638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41480.log.gz 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27812.log 2024-09-18T14:33:56.430 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26125.log 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48424.log 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25905.log.gz 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27812.log: /var/log/ceph/ceph-client.admin.26125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27812.log.gz 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26125.log.gz 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31864.log 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64823.log 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48424.log.gz 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31864.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31864.log.gz 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37990.log 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32512.log 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64823.log: /var/log/ceph/ceph-client.admin.37990.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64823.log.gz 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37990.log.gz 2024-09-18T14:33:56.431 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27758.log 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33241.log 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44291.log 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27758.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32512.log.gz 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27758.log.gz 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26979.log 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33241.log.gz 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61089.log 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44291.log.gz 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 /var/log/ceph/ceph-client.admin.26979.log: --verbose -- /var/log/ceph/ceph-client.admin.49936.log 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26979.log.gz 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61089.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61089.log.gz 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46128.log 2024-09-18T14:33:56.432 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63979.log 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49936.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49936.log.gz 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46074.log 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46128.log.gz 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63979.log.gz 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27515.log 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46074.log.gz 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35411.log 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.27515.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.32107.log 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27515.log.gz 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35411.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35411.log.gz 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52313.log 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66823.log 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32107.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32107.log.gz 2024-09-18T14:33:56.433 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50989.log 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52313.log: /var/log/ceph/ceph-client.admin.66823.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52313.log.gz 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66823.log.gz 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38449.log 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42398.log 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59305.log 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50989.log.gz 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38449.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38449.log.gz 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42398.log.gz 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29108.log 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-18T14:33:56.434 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59305.log.gz 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33818.log 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29108.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29108.log.gz 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59251.log 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33818.log.gz 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63628.log 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59251.log.gz 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40885.log 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64195.log 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63628.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63628.log.gz 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45966.log 2024-09-18T14:33:56.435 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40885.log.gz 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37207.log 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64195.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64195.log.gz 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52955.log 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45966.log.gz 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37207.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37207.log.gz 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30649.log 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52955.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52955.log.gz 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38422.log 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37774.log 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30649.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30649.log.gz 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52205.log 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38422.log.gz 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52394.log 2024-09-18T14:33:56.436 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37774.log.gz 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52205.log.gz 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49666.log 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52394.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52394.log.gz 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49423.log 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44318.log 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49666.log: /var/log/ceph/ceph-client.admin.49423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60251.log 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49666.log.gz -- replaced with /var/log/ceph/ceph-client.admin.49423.log.gz 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31378.log 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44318.log.gz 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60251.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60251.log.gz 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48316.log 2024-09-18T14:33:56.437 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31378.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31378.log.gz 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29596.log 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39178.log 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48316.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48316.log.gz 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33592.log 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29596.log.gz 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41642.log 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39178.log.gz 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64499.log 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33592.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33592.log.gz 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41642.log.gz 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56686.log 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64499.log.gz 2024-09-18T14:33:56.438 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35978.log 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56686.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.56686.log.gz 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50098.log 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47641.log 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35978.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35978.log.gz 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36774.log 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50098.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50098.log.gz 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48856.log 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47641.log.gz 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53053.log 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36774.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36774.log.gz 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37423.log 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48856.log.gz 2024-09-18T14:33:56.439 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53053.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.53053.log.gz--verbose 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.36005.log 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26361.log 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37423.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37423.log.gz 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36005.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.65959.log 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.36005.log.gz 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52668.log 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26361.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26361.log.gz 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31945.log 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65959.log.gz 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52668.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37504.log 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52668.log.gz 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31999.log 2024-09-18T14:33:56.440 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31945.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31945.log.gz 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37504.log.gz 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40372.log 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64742.log 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31999.log.gz 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57928.log 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40372.log.gz 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48505.log 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64742.log.gz 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57928.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57928.log.gz 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47803.log 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45615.log 2024-09-18T14:33:56.441 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48505.log.gz 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47803.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47803.log.gz 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35789.log 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45615.log: /var/log/ceph/ceph-client.admin.40399.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.45615.log.gz--verbose 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.63061.log 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43454.log 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35789.log.gz 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63061.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40210.log 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63061.log.gz 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57172.log 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43454.log: /var/log/ceph/ceph-client.admin.40210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43454.log.gz 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40210.log.gz 2024-09-18T14:33:56.442 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47317.log 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32242.log 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57172.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57172.log.gz 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 /var/log/ceph/ceph-client.admin.47317.log: --verbose -- /var/log/ceph/ceph-client.admin.34547.log 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47317.log.gz 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32242.log.gz 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48019.log 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30541.log 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34547.log: /var/log/ceph/ceph-client.admin.48019.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34547.log.gz 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32620.log 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48019.log.gz 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38584.log 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30541.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30541.log.gz 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45318.log 2024-09-18T14:33:56.443 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29677.log 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32620.log.gz 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38584.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38584.log.gz 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61927.log 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45318.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45318.log.gz 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29677.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29677.log.gz 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35735.log 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26334.log 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61927.log.gz 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35735.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.35735.log.gz 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.45831.log 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57496.log 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26334.log.gz 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46101.log 2024-09-18T14:33:56.444 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45831.log: /var/log/ceph/ceph-client.admin.57496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33710.log 2024-09-18T14:33:56.445 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45831.log.gz 2024-09-18T14:33:56.445 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57496.log.gz 2024-09-18T14:33:56.445 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33791.log 2024-09-18T14:33:56.445 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46101.log: /var/log/ceph/ceph-client.admin.33710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46101.log.gz 2024-09-18T14:33:56.445 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33710.log.gz 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58954.log 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33791.log.gz 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28892.log 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58333.log 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58954.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58954.log.gz 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42236.log 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28892.log.gz 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33268.log 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58333.log: /var/log/ceph/ceph-client.admin.42236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33457.log 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42236.log.gz 2024-09-18T14:33:56.507 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58333.log.gz 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51691.log 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33268.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33268.log.gz 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33457.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33457.log.gz 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45291.log 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51691.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32647.log 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51691.log.gz 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61170.log 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26507.log 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45291.log.gz/var/log/ceph/ceph-client.admin.32647.log: 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32647.log.gz 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61170.log.gz 2024-09-18T14:33:56.508 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42128.log 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29569.log 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26507.log.gz 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37180.log 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42128.log.gz 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25914.log 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39886.log 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29569.log: /var/log/ceph/ceph-client.admin.37180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37180.log.gz 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29569.log.gz 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56389.log 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25914.log.gz 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52446.log 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39886.log.gz 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53353.log 2024-09-18T14:33:56.509 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56389.log.gz 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52954.log 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50773.log 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53353.log: /var/log/ceph/ceph-client.admin.52954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66337.log 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53353.log.gz 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52954.log.gz 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39613.log 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50773.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50773.log.gz 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45723.log 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr: 94.2% -- replaced with /var/log/ceph/ceph-client.admin.52446.log.gz 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56254.log 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66337.log.gz 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39613.log: gzip /var/log/ceph/ceph-client.admin.45723.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36720.log 2024-09-18T14:33:56.510 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39613.log.gz 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45723.log.gz 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53269.log 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56254.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56254.log.gz 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63358.log 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36720.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36720.log.gz 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35222.log 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53269.log.gz 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51907.log 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63358.log: gzip /var/log/ceph/ceph-client.admin.35222.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.47128.log 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63358.log.gz 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35222.log.gz 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44048.log 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51907.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51907.log.gz 2024-09-18T14:33:56.511 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65175.log 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47128.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47128.log.gz 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52096.log 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44048.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44048.log.gz 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37126.log 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65175.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65175.log.gz 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41696.log 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65554.log 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52096.log.gz 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37126.log.gz 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30946.log 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41696.log.gz 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65554.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65554.log.gz 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56713.log 2024-09-18T14:33:56.512 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46155.log 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30946.log.gz 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64141.log 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56713.log.gz 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46155.log.gz 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58846.log 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34412.log 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64141.log.gz 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58846.log.gz 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34709.log 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66634.log 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34412.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34412.log.gz 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28487.log 2024-09-18T14:33:56.513 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34709.log.gz 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53323.log 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66634.log.gz 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28055.log 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28487.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28487.log.gz 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57712.log 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53323.log: /var/log/ceph/ceph-client.admin.28055.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.44859.log 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53323.log.gz 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28055.log.gz 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49504.log 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57712.log.gz 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44859.log.gz 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38395.log 2024-09-18T14:33:56.514 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49504.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49504.log.gz 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35006.log 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38395.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38395.log.gz 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65824.log 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59197.log 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38908.log 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65824.log.gz 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63169.log 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35006.log.gz 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59197.log.gz 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33737.log 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38908.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38908.log.gz 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52340.log 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63169.log.gz 2024-09-18T14:33:56.515 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29432.log 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33737.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33737.log.gz 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 /var/log/ceph/ceph-client.admin.52340.log: --verbose -- /var/log/ceph/ceph-client.admin.40642.log 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52340.log.gz 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27110.log 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29432.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29432.log.gz 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38503.log 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40642.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40642.log.gz 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27110.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41020.log 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27110.log.gz 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44967.log 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38503.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59224.log 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38503.log.gz 2024-09-18T14:33:56.516 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41020.log: 0.0%/var/log/ceph/ceph-client.admin.44967.log: -- replaced with /var/log/ceph/ceph-client.admin.41020.log.gzgzip 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.40345.log 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44967.log.gz 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62575.log 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59224.log.gz 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40345.log: /var/log/ceph/ceph-client.admin.57226.log 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40345.log.gz 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28379.log 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62575.log.gz 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33511.log 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57226.log.gz 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35843.log 2024-09-18T14:33:56.517 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28379.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28379.log.gz 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44129.log 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33511.log.gz 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46426.log 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35843.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35843.log.gz 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46480.log 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44129.log.gz 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46426.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46426.log.gz 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46669.log 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46480.log.gz 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38611.log 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41831.log 2024-09-18T14:33:56.518 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46669.log.gz 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62224.log 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38611.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38611.log.gz 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41831.log.gz 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22290.log 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31162.log 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62224.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62224.log.gz 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65716.log 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.22290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22290.log.gz 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40075.log 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31162.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31162.log.gz 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31324.log 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65716.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65716.log.gz 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31108.log 2024-09-18T14:33:56.519 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40075.log.gz 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55741.log 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31324.log.gz 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60305.log 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31108.log: /var/log/ceph/ceph-client.admin.55741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40750.log 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31108.log.gz 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55741.log.gz 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.60305.log: -- /var/log/ceph/ceph-client.admin.60467.log 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60305.log.gz 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35330.log 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40750.log: /var/log/ceph/ceph-client.admin.60467.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40750.log.gz 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47209.log 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60467.log.gz 2024-09-18T14:33:56.520 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51637.log 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35330.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35330.log.gz 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41264.log 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47209.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47209.log.gz 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56200.log 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51637.log.gz 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28136.log 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41264.log: gzip -5 --verbose 0.0%-- /var/log/ceph/ceph-client.admin.51988.log 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41264.log.gz 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56200.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56200.log.gz 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42590.log 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28136.log.gz 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60792.log 2024-09-18T14:33:56.521 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51988.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51988.log.gz 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45237.log 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42590.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42590.log.gz 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33430.log 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60792.log.gz 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32269.log 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45237.log.gz 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37315.log 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33430.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33430.log.gz 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53215.log 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32269.log.gz 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47911.log 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37315.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37315.log.gzgzip 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.60981.log 2024-09-18T14:33:56.522 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51205.log 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47911.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53215.log.gz 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47911.log.gz 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.60981.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39913.log 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60981.log.gz 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49450.log 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51205.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51205.log.gz 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43535.log 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39913.log.gz 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40804.log 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49450.log.gz 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43535.log: /var/log/ceph/ceph-client.admin.48991.log 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43535.log.gz 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44561.log 2024-09-18T14:33:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40804.log.gz 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48991.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48991.log.gz 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59629.log 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44561.log.gz 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41588.log 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59629.log.gz 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46993.log 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48964.log 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41588.log.gz 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64986.log 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46993.log.gz 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43049.log 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48964.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48964.log.gz 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43481.log 2024-09-18T14:33:56.524 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64986.log.gz 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46047.log 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43049.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43049.log.gz 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57442.log 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43481.log.gz 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36356.log 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46047.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46047.log.gz 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33683.log 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57442.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57442.log.gz 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63304.log 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36356.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36356.log.gz 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33683.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31918.log 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33683.log.gz 2024-09-18T14:33:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41777.log 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63304.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63304.log.gz 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52069.log 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31918.log.gz 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53242.log 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41777.log.gz 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56740.log 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52069.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52069.log.gz 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63682.log 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53242.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53242.log.gz 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56740.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56740.log.gz 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44507.log 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63682.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63682.log.gz 2024-09-18T14:33:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47938.log 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44507.log.gz 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58792.log 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47938.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47938.log.gz 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62629.log 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58792.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58792.log.gz 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33322.log 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47776.log 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62629.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62629.log.gz 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37963.log 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33322.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33322.log.gz 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57523.log 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47776.log.gz 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37963.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22225.log 2024-09-18T14:33:56.527 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37963.log.gz 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57523.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57523.log.gz 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51583.log 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57280.log 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.22225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38692.log 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22225.log.gz 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66499.log 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51583.log.gz 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57280.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57280.log.gz 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32809.log 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38692.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38692.log.gz 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42317.log 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66499.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66499.log.gz 2024-09-18T14:33:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61305.log /var/log/ceph/ceph-client.admin.32809.log: 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32809.log.gz 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44778.log 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42317.log.gz 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61305.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58036.log 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61305.log.gz 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27299.log 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44778.log.gz 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39805.log 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58036.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58036.log.gz 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62845.log 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27299.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.27299.log.gz 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.51124.log 2024-09-18T14:33:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39805.log.gz 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34007.log 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62845.log.gz 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36747.log 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51124.log.gz 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56038.log 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34007.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34007.log.gz 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27434.log 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36747.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36747.log.gz 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61332.log 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56038.log.gz 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36531.log 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27434.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27434.log.gz 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57874.log 2024-09-18T14:33:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61332.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61332.log.gz 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47506.log 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36531.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36531.log.gz/var/log/ceph/ceph-client.admin.57874.log: 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37450.log 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57874.log.gz 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50962.log 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47506.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47506.log.gz 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33646.log 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37450.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37450.log.gz 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50962.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50962.log.gz 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28649.log 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33646.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33646.log.gz 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32350.log 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47533.log 2024-09-18T14:33:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61440.log 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28649.log.gz 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32350.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32350.log.gz 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52422.log 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47533.log.gz 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61440.log: /var/log/ceph/ceph-client.admin.63655.log 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61440.log.gz 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31000.log 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52422.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52422.log.gz 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30271.log 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63655.log.gz 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31000.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45156.log 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31000.log.gz 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63412.log 2024-09-18T14:33:56.532 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30271.log.gz 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45156.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44886.log 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45156.log.gz 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41102.log 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63412.log: 0.0%gzip -5 -- replaced with /var/log/ceph/ceph-client.admin.63412.log.gz--verbose 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:-- /var/log/ceph/ceph-client.admin.55930.log 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44886.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44886.log.gz 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66580.log 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41102.log.gz 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50206.log 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55930.log.gz 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44697.log 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66580.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66580.log.gz 2024-09-18T14:33:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50206.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50206.log.gz 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29947.log 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40993.log 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44697.log.gz 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33160.log 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53620.log 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40993.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29947.log.gz 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40993.log.gz 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37369.log 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33160.log.gz 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53620.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53620.log.gz 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61062.log 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57820.log 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37369.log.gz 2024-09-18T14:33:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61062.log.gz 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42725.log 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58495.log 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57820.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57820.log.gz 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42725.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.42725.log.gz/var/log/ceph/ceph-client.admin.57793.log 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58495.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58495.log.gz 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26925.log 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 /var/log/ceph/ceph-client.admin.57793.log: --verbose -- /var/log/ceph/ceph-client.admin.38476.log 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57793.log.gz 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58711.log 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26925.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26925.log.gz 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38476.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38476.log.gz 2024-09-18T14:33:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29000.log 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55606.log 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58711.log.gz 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43643.log 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29000.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29000.log.gz 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55606.log: /var/log/ceph/ceph-client.admin.66553.log 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55606.log.gz 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43643.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35141.log 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43643.log.gz 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52800.log 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66553.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66553.log.gz 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35141.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35141.log.gz 2024-09-18T14:33:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36491.log 2024-09-18T14:33:56.584 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55822.log 2024-09-18T14:33:56.585 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52800.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52800.log.gz 2024-09-18T14:33:56.585 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27407.log 2024-09-18T14:33:56.585 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36491.log.gz 2024-09-18T14:33:56.585 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.55822.log: -- /var/log/ceph/ceph-client.admin.52844.log 2024-09-18T14:33:56.585 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55822.log.gz 2024-09-18T14:33:56.585 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62143.log 2024-09-18T14:33:56.585 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27407.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27407.log.gz 2024-09-18T14:33:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42509.log 2024-09-18T14:33:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52844.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52844.log.gz 2024-09-18T14:33:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.62143.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.60819.log 2024-09-18T14:33:56.588 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62143.log.gz 2024-09-18T14:33:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26616.log 2024-09-18T14:33:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42509.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42509.log.gz 2024-09-18T14:33:56.588 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36990.log 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60819.log.gz 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66904.log 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26616.log.gz 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.36990.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41372.log 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36990.log.gz 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66904.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66904.log.gz 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60521.log 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36666.log 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41372.log.gz 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33565.log 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45912.log 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60521.log.gz 2024-09-18T14:33:56.589 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36666.log: /var/log/ceph/ceph-client.admin.33565.log: gzip -5 --verbose 0.0%-- -- replaced with /var/log/ceph/ceph-client.admin.36666.log.gz/var/log/ceph/ceph-client.admin.42995.log 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33565.log.gz 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33953.log 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45912.log.gz 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48694.log 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42995.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42995.log.gz 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51664.log /var/log/ceph/ceph-client.admin.33953.log: 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33953.log.gz 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58441.log 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48694.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48694.log.gz 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31972.log 2024-09-18T14:33:56.590 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51664.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51664.log.gz 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58441.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58441.log.gz 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66958.log 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40912.log 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64688.log 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31972.log.gz 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66958.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66958.log.gz 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61359.log 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40912.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40912.log.gz 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64688.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61251.log 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64688.log.gz 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26398.log 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61359.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61359.log.gz 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43427.log 2024-09-18T14:33:56.591 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61251.log: /var/log/ceph/ceph-client.admin.26398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61251.log.gz 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59413.log 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26398.log.gz 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61116.log 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43427.log.gz 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52514.log 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59413.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.59413.log.gz-5 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.47182.log 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61116.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61116.log.gz 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61981.log 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52514.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.52514.log.gz-5 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.49585.log 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47182.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47182.log.gz 2024-09-18T14:33:56.592 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61981.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61981.log.gz 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40831.log 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42155.log 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49585.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49585.log.gz 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55549.log 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40831.log.gz 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34601.log 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42155.log.gz 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48370.log 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55549.log.gz 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39532.log 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34601.log: /var/log/ceph/ceph-client.admin.48370.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48370.log.gz 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49396.log 2024-09-18T14:33:56.593 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43805.log 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39532.log: /var/log/ceph/ceph-client.admin.49396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39532.log.gz 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49396.log.gz 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29785.log 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45399.log 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43805.log.gz 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40696.log 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29785.log.gz 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31675.log 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45399.log.gz 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40696.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40696.log.gz 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41237.log 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31675.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31675.log.gz 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55660.log 2024-09-18T14:33:56.594 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34061.log 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41237.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41237.log.gz 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46777.log 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55660.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55660.log.gz 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.34061.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.65365.log 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34061.log.gz 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58873.log 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46777.log.gz 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65365.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65365.log.gz 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63466.log 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58873.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58873.log.gz 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52756.log 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63466.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63466.log.gz 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39670.log 2024-09-18T14:33:56.595 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26018.log 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52756.log.gz 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25801.log 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39670.log.gz 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58900.log 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26018.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26018.log.gz 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25801.log.gz 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47344.log 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42452.log 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58900.log.gz 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32971.log 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47344.log.gz 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28595.log 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42452.log.gz 2024-09-18T14:33:56.596 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39097.log 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32971.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32971.log.gz 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25821.log 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28595.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28595.log.gz 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39097.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45129.log 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39097.log.gz 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25821.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25821.log.gz 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34574.log 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose /var/log/ceph/ceph-client.admin.45129.log: -- /var/log/ceph/ceph-client.admin.59143.log 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45129.log.gz 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57631.log 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34574.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34574.log.gz 2024-09-18T14:33:56.597 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58927.log 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59143.log.gz 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30919.log 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57631.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57631.log.gz 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37801.log 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58927.log: /var/log/ceph/ceph-client.admin.30919.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58927.log.gz 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30919.log.gz 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35114.log 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37801.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37801.log.gz 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45426.log 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35114.log.gz 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31513.log 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45426.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64033.log 2024-09-18T14:33:56.598 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45426.log.gz 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26821.log 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31513.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31513.log.gz 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59818.log 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64033.log.gz 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50179.log 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26821.log.gz 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59818.log.gz 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49099.log 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50179.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50179.log.gz 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66283.log 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44805.log 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49099.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49099.log.gz 2024-09-18T14:33:56.599 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63763.log 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66283.log.gz 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66364.log 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44805.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44805.log.gz 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63763.log.gz 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46588.log 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66364.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66364.log.gz 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27596.log 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46588.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46588.log.gz 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.21970.log 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27596.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27596.log.gz 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51151.log 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.21970.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.21970.log.gz 2024-09-18T14:33:56.600 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45885.log 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31567.log 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51151.log.gz 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50638.log 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45885.log: /var/log/ceph/ceph-client.admin.31567.log: gzip 0.0%-5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.45885.log.gz-- 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.log 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31567.log.gz 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33403.log 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50638.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50638.log.gz 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42425.log 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29974.log 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33403.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33403.log.gz 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42425.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55714.log 2024-09-18T14:33:56.601 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42425.log.gz 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59764.log 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29974.log.gz 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61818.log 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55714.log.gz 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63817.log 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59764.log.gz 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57550.log 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64249.log 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61818.log.gz 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57550.log: /var/log/ceph/ceph-client.admin.63817.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57550.log.gz 2024-09-18T14:33:56.602 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63817.log.gz 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48343.log 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64249.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64249.log.gz 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48640.log 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48343.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48343.log.gz 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33106.log 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48640.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48640.log.gz 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50125.log 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33106.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33106.log.gz 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47263.log 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50125.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50125.log.gz 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35492.log 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32728.log 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47263.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47263.log.gz 2024-09-18T14:33:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35492.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35492.log.gz 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42644.log 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42914.log 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32728.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32728.log.gz 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42644.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42644.log.gz 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48478.log 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35249.log 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42914.log.gz 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46398.log 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41345.log 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48478.log.gz 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35249.log: /var/log/ceph/ceph-client.admin.46398.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46398.log.gz 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35249.log.gz 2024-09-18T14:33:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66256.log 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49315.log 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41345.log.gz 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28028.log 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49315.log: /var/log/ceph/ceph-client.admin.66256.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49315.log.gz 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34088.log 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%gzip -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.66256.log.gz/var/log/ceph/ceph-client.admin.26739.log 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34088.log: /var/log/ceph/ceph-client.admin.28028.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34088.log.gz 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28028.log.gz 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39697.log 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26739.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26739.log.gz 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30703.log 2024-09-18T14:33:56.605 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39697.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60061.log 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39697.log.gz 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30352.log 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30703.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30703.log.gz 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60061.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60061.log.gz 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42833.log 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30352.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30352.log.gz 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59791.log 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27461.log 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32782.log 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59791.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42833.log.gz 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59791.log.gz 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27461.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27461.log.gz 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28811.log 2024-09-18T14:33:56.606 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32782.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32782.log.gz 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52778.log 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28811.log.gz 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49801.log 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48559.log 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52778.log.gz 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27164.log 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49801.log.gz 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48559.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48559.log.gz 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41210.log 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27164.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27164.log.gz 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47965.log 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41210.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41210.log.gz 2024-09-18T14:33:56.607 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42371.log 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47965.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47965.log.gz 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56146.log 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42371.log.gz 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39559.log 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56146.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56146.log.gz 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52123.log 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39559.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65635.log 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39559.log.gz 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57037.log 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52123.log.gz 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32674.log 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65635.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65635.log.gz 2024-09-18T14:33:56.608 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56443.log 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57037.log: 0.0%/var/log/ceph/ceph-client.admin.32674.log: -- replaced with /var/log/ceph/ceph-client.admin.57037.log.gz 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32674.log.gz 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35384.log 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62548.log 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56443.log.gz 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38854.log 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35384.log: /var/log/ceph/ceph-client.admin.62548.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35384.log.gz 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62548.log.gz 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63223.log 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29920.log 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38854.log: /var/log/ceph/ceph-client.admin.63223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38854.log.gz 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63223.log.gz 2024-09-18T14:33:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59953.log 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29920.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58819.log 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29920.log.gz 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29893.log 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59953.log.gz 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39778.log 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58819.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58819.log.gz 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29893.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29893.log.gz 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31756.log 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39778.log.gz 2024-09-18T14:33:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36275.log 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31756.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31756.log.gz 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30838.log 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36275.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36275.log.gz 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30298.log 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30838.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30838.log.gz 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58765.log 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28838.log 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30298.log: /var/log/ceph/ceph-client.admin.58765.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30298.log.gz 2024-09-18T14:33:56.660 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.58765.log.gz 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.40588.log 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22396.log 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28838.log: /var/log/ceph/ceph-client.admin.40588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62818.log 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28838.log.gz 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40588.log.gz 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49342.log 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.22396.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22396.log.gz 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62818.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62818.log.gz 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66796.log 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53107.log 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49342.log.gz 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46561.log 2024-09-18T14:33:56.661 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29515.log 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66796.log.gz 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53107.log: /var/log/ceph/ceph-client.admin.46561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53107.log.gz 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38962.log 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46561.log.gz 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64959.log 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29515.log.gz 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60738.log 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61035.log 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64959.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38962.log.gz 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64959.log.gz 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.22250.log 2024-09-18T14:33:56.662 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60738.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60738.log.gz 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48937.log 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61035.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61035.log.gz 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.22250.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.22250.log.gz 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47371.log 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48937.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29623.log 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48937.log.gz 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47371.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47371.log.gz 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32323.log 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47425.log 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29623.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29623.log.gz 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64526.log 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32323.log.gz 2024-09-18T14:33:56.663 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46804.log 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47425.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47425.log.gz 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51259.log 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64526.log.gz 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46804.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46804.log.gz 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46317.log 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51259.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51259.log.gz 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64418.log 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46317.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46317.log.gz 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49990.log 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27353.log 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64418.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64418.log.gz 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35762.log 2024-09-18T14:33:56.664 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49990.log.gz 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42074.log 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27353.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27353.log.gz 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35762.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35762.log.gz 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39124.log 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64364.log 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42074.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42074.log.gz 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60656.log 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39124.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39124.log.gz 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.64364.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.50557.log 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64364.log.gz 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60656.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60656.log.gz 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48586.log 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50557.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50557.log.gz 2024-09-18T14:33:56.665 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66931.log 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34817.log 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30784.log 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48586.log.gz 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66931.log: /var/log/ceph/ceph-client.admin.34817.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.66931.log.gz-5 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.31486.log 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34817.log.gz 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63277.log 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30784.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30784.log.gz 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31486.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40777.log 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31486.log.gz 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58279.log 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64905.log 2024-09-18T14:33:56.666 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63277.log.gz 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40777.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40777.log.gz 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58279.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25913.log 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58279.log.gz 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64905.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64905.log.gz 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62332.log 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33926.log 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25913.log.gz 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38044.log 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62332.log.gz 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44156.log 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33926.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33926.log.gz 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50368.log 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38044.log: /var/log/ceph/ceph-client.admin.44156.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38044.log.gz 2024-09-18T14:33:56.667 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44156.log.gz 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51367.log 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51421.log 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50368.log.gz 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51367.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51367.log.gz 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31351.log 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51421.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51421.log.gz 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57955.log 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31351.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64932.log 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31351.log.gz 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67120.log 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57955.log.gz 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29135.log 2024-09-18T14:33:56.668 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64932.log.gz 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41291.log 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.67120.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67120.log.gz 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49180.log 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29135.log.gz 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67095.log 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41291.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41291.log.gz 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49180.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49180.log.gz 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53516.log 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.67095.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.67095.log.gz 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53462.log 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53516.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53516.log.gz 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50908.log 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64445.log 2024-09-18T14:33:56.669 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53462.log.gz 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65013.log 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43670.log 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64445.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50908.log.gz 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64445.log.gz 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65013.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65013.log.gz 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43400.log 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43670.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43670.log.gz 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62170.log 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40507.log 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 /var/log/ceph/ceph-client.admin.43400.log: --verbose -- /var/log/ceph/ceph-client.admin.39940.log 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62170.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62170.log.gz 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43400.log.gz 2024-09-18T14:33:56.670 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40507.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40507.log.gz 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59710.log 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44102.log 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65851.log 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59710.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39940.log.gz 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59710.log.gz 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44102.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44102.log.gz 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32890.log 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65851.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65851.log.gz 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39832.log 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38773.log 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.32890.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.39478.log 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39832.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32890.log.gz 2024-09-18T14:33:56.671 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62467.log 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38773.log.gz 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64715.log 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39478.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39478.log.gz 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29270.log 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62467.log.gz 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34898.log 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64715.log.gz 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28217.log 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29270.log.gz 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34898.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47614.log 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34898.log.gz 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33538.log 2024-09-18T14:33:56.672 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47614.log: /var/log/ceph/ceph-client.admin.28217.log: gzip -5 --verbose -- 0.0%/var/log/ceph/ceph-client.admin.35816.log -- replaced with /var/log/ceph/ceph-client.admin.47614.log.gz 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28217.log.gz 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33538.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33538.log.gz 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52558.log 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49828.log 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35816.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35816.log.gz 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53114.log 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52558.log.gz 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34223.log 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49828.log.gz 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40939.log 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53114.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53114.log.gz 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34223.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34223.log.gz 2024-09-18T14:33:56.673 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43238.log 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29216.log 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40939.log.gz 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37666.log 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43238.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43238.log.gz 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37477.log 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29216.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.29216.log.gz 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64006.log 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37666.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37666.log.gz 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37477.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37477.log.gz 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60440.log 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61846.log 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64006.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64006.log.gz 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38179.log 2024-09-18T14:33:56.674 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60440.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60440.log.gz 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48748.log 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61846.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61846.log.gz 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38179.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44453.log 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38179.log.gz 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46507.log 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48748.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48748.log.gz 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36693.log 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44453.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44453.log.gz 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46507.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30676.log 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46507.log.gz 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34925.log 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36693.log.gz 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31297.log 2024-09-18T14:33:56.675 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30676.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30676.log.gz 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35654.log 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34925.log.gz 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31297.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66013.log 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31297.log.gz 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60224.log 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35654.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35654.log.gz 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56767.log 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66013.log.gz 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34979.log 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60224.log.gz 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47884.log 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56767.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56767.log.gz 2024-09-18T14:33:56.676 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34979.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34979.log.gz 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44399.log 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62116.log 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47884.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47884.log.gz 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42482.log 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44399.log.gz 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45075.log 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62116.log.gz 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41885.log 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42482.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42482.log.gz 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56416.log 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45075.log: 0.0%/var/log/ceph/ceph-client.admin.41885.log: -- replaced with /var/log/ceph/ceph-client.admin.45075.log.gz 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51853.log 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41885.log.gz 2024-09-18T14:33:56.677 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44372.log 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56416.log: /var/log/ceph/ceph-client.admin.51853.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56416.log.gz 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65932.log 0.0% 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51853.log.gz 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30325.log 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44372.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44372.log.gz 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65932.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65932.log.gz 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41669.log 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30325.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30325.log.gz 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43103.log 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41669.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41669.log.gz 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63385.log 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43103.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43103.log.gz 2024-09-18T14:33:56.678 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41966.log 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47695.log 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63385.log.gz 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38152.log 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41966.log.gz 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30730.log 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47695.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47695.log.gz 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38152.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38152.log.gz 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39859.log 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63547.log 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30730.log.gz 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39859.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39859.log.gz 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37342.log 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.63547.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.45588.log 2024-09-18T14:33:56.679 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63547.log.gz 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34358.log 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37342.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37342.log.gz 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61900.log 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45588.log.gz 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25749.log 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34358.log: gzip 0.0%-5 -- replaced with /var/log/ceph/ceph-client.admin.34358.log.gz--verbose -- 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46831.log 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61900.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61900.log.gz 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.25749.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.25749.log.gz 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27785.log 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46831.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46831.log.gz 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41723.log 2024-09-18T14:33:56.680 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.27785.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.27785.log.gz 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51880.log 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28271.log 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41723.log.gz 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49126.log 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51880.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51880.log.gz 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42020.log 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28271.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28271.log.gz 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28082.log 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49126.log.gz 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42020.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42020.log.gz 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62035.log 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66715.log 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28082.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28082.log.gz 2024-09-18T14:33:56.681 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62386.log 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62035.log.gz 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.66715.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.66715.log.gz 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47155.log 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63331.log 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48802.log 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47155.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62386.log.gz 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47155.log.gz 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36194.log 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63331.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63331.log.gz 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48802.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48802.log.gz 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32863.log 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36194.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36194.log.gz 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45777.log 2024-09-18T14:33:56.682 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32863.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32863.log.gz 2024-09-18T14:33:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33980.log 2024-09-18T14:33:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63871.log 2024-09-18T14:33:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61575.log 2024-09-18T14:33:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45777.log: /var/log/ceph/ceph-client.admin.33980.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45777.log.gz 2024-09-18T14:33:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33980.log.gz 2024-09-18T14:33:56.683 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61413.log 2024-09-18T14:33:56.683 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63871.log.gz 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61575.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52712.log 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61575.log.gz 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52957.log 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61413.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61413.log.gz 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52712.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52712.log.gz 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44345.log 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52957.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52957.log.gz 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30865.log 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26118.log 2024-09-18T14:33:56.711 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44345.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44345.log.gz 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31810.log 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.30865.log: /var/log/ceph/ceph-client.admin.26118.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.30865.log.gz 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26118.log.gz 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40669.log 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31810.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41939.log 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31810.log.gz 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35168.log 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41939.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41939.log.gz 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32377.log 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40669.log.gz 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62305.log 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35168.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35168.log.gz 2024-09-18T14:33:56.712 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37828.log 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32377.log.gz 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62764.log 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62305.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62305.log.gz 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37828.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37828.log.gz 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47047.log 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57658.log 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62764.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62764.log.gzgzip 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.57604.log 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47047.log: /var/log/ceph/ceph-client.admin.57658.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47047.log.gz 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45561.log 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57658.log.gz 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65310.log 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57604.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57604.log.gz 2024-09-18T14:33:56.713 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.45561.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.45561.log.gz 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50854.log 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63196.log 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.65310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65310.log.gz 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61008.log 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60927.log 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50854.log.gz 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61008.log: /var/log/ceph/ceph-client.admin.63196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31189.log 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61008.log.gz 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63196.log.gz 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33133.log 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60927.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60927.log.gz 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31189.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.31189.log.gz 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48289.log 2024-09-18T14:33:56.714 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55333.log 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.33133.log: /var/log/ceph/ceph-client.admin.48289.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.33133.log.gz 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48289.log.gz 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41399.log 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39724.log 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55333.log.gz 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46534.log 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41399.log.gz 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38125.log 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39724.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.39724.log.gz 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46534.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46534.log.gz 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26226.log 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60494.log 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.38125.log: /var/log/ceph/ceph-client.admin.26226.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38125.log.gz 2024-09-18T14:33:56.715 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43751.log 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26226.log.gz 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34277.log 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.60494.log: gzip -5 --verbose /var/log/ceph/ceph-client.admin.43751.log: -- /var/log/ceph/ceph-client.admin.55903.log 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60494.log.gz 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43751.log.gz 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34277.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43778.log 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34277.log.gz 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32539.log 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.55903.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.55903.log.gz 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51502.log 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43778.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43778.log.gz 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32539.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35087.log 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32539.log.gz 2024-09-18T14:33:56.716 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40453.log 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29731.log 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51502.log.gz 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35087.log.gz 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40453.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32026.log 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40453.log.gz 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28622.log 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.29731.log: 0.0%/var/log/ceph/ceph-client.admin.32026.log: -- replaced with /var/log/ceph/ceph-client.admin.29731.log.gz 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31081.log 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32026.log.gz 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60765.log 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64087.log 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.31081.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28622.log.gz 2024-09-18T14:33:56.717 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0%/var/log/ceph/ceph-client.admin.60765.log: -- replaced with /var/log/ceph/ceph-client.admin.31081.log.gz 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60765.log.gz 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46344.log 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40480.log 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64087.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64087.log.gz 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46344.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46344.log.gzgzip 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:-5 --verbose -- /var/log/ceph/ceph-client.admin.37612.log 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26824.log 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.40480.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.40480.log.gz 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.37612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52602.log 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.37612.log.gz 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51934.log 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26824.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26824.log.gz 2024-09-18T14:33:56.718 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.52602.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.52602.log.gz 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39994.log 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62062.log 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51934.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51934.log.gz 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36558.log 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.39994.log: gzip -5 0.0%--verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39994.log.gz 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.51718.log 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62062.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62062.log.gz 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.26126.log 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36558.log.gz 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.51718.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.62953.log 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.51718.log.gz 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32161.log 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.26126.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.26126.log.gz 2024-09-18T14:33:56.719 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62953.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62953.log.gz 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41129.log 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.32161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49369.log 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.32161.log.gz 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50314.log 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.41129.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.41129.log.gz 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42290.log 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.49369.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.49369.log.gz 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53296.log 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50314.log.gz 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42290.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42290.log.gz 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56902.log 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.53296.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.53296.log.gz 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48262.log 2024-09-18T14:33:56.720 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56902.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56902.log.gz 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58630.log 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.65662.log 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48262.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48262.log.gz 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47560.log 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.58630.log: gzip /var/log/ceph/ceph-client.admin.65662.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.61521.log 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.58630.log.gz 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.65662.log.gz 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47560.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59521.log 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47560.log.gz 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.61521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28946.log 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.61521.log.gz 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.59521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42536.log 2024-09-18T14:33:56.721 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.59521.log.gz 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38314.log 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.28946.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.28946.log.gz 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34871.log 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42536.log: 0.0%/var/log/ceph/ceph-client.admin.38314.log: -- replaced with /var/log/ceph/ceph-client.admin.42536.log.gz 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.38314.log.gz 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44615.log 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.34871.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.34871.log.gz 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.44615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64276.log 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.44615.log.gz 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36221.log 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56929.log 2024-09-18T14:33:56.722 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.64276.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.64276.log.gz 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.36221.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.36221.log.gz 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35033.log 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60115.log 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56929.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56929.log.gz 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63520.log 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.35033.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.35033.log.gz 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip /var/log/ceph/ceph-client.admin.60115.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.57361.log 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.60115.log.gz 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63520.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.63520.log.gz 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63088.log 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.57361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43616.log 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.57361.log.gz 2024-09-18T14:33:56.723 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50530.log 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.63088.log: 0.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.63088.log.gz-5 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:--verbose -- /var/log/ceph/ceph-client.admin.48100.log 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.43616.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.43616.log.gz 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.50530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62008.log 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.50530.log.gz 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.48100.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.48100.log.gz 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46966.log 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62008.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62008.log.gz 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47857.log 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42344.log 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.46966.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.46966.log.gz 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.47857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.62197.log 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.47857.log.gz 2024-09-18T14:33:56.724 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.42344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56119.log 2024-09-18T14:33:56.725 INFO:teuthology.orchestra.run.smithi016.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.42344.log.gz 2024-09-18T14:33:56.725 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.62197.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.62197.log.gz 2024-09-18T14:33:56.725 INFO:teuthology.orchestra.run.smithi016.stderr:/var/log/ceph/ceph-client.admin.56119.log: 0.0% -- replaced with /var/log/ceph/ceph-client.admin.56119.log.gz 2024-09-18T14:33:56.848 INFO:teuthology.orchestra.run.smithi016.stderr: 28.2% -- replaced with /var/log/ceph/ceph-client.admin.34601.log.gz 2024-09-18T14:33:56.947 INFO:teuthology.orchestra.run.smithi016.stderr: 96.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-18T14:33:57.520 INFO:teuthology.orchestra.run.smithi016.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-18T14:33:57.964 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-18T14:33:59.563 INFO:teuthology.orchestra.run.smithi016.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-18T14:34:00.669 INFO:teuthology.orchestra.run.smithi016.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-18T14:34:02.151 INFO:teuthology.orchestra.run.smithi016.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-18T14:34:22.155 INFO:teuthology.orchestra.run.smithi016.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-18T14:34:26.127 INFO:teuthology.orchestra.run.smithi016.stderr: 94.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-18T14:36:56.909 INFO:teuthology.orchestra.run.smithi016.stderr: 90.2% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-18T14:37:47.885 INFO:teuthology.orchestra.run.smithi016.stderr: 90.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-18T14:37:47.887 INFO:teuthology.orchestra.run.smithi016.stderr: 2024-09-18T14:37:47.887 INFO:teuthology.orchestra.run.smithi016.stderr:real 3m51.876s 2024-09-18T14:37:47.887 INFO:teuthology.orchestra.run.smithi016.stderr:user 4m56.994s 2024-09-18T14:37:47.887 INFO:teuthology.orchestra.run.smithi016.stderr:sys 0m10.823s 2024-09-18T14:37:47.887 INFO:tasks.ceph:Archiving logs... 2024-09-18T14:37:47.888 DEBUG:teuthology.misc:Transferring archived files from smithi016:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080/remote/smithi016/log 2024-09-18T14:37:47.889 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-18T14:38:41.178 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-18T14:38:41.194 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-18T14:38:41.195 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-18T14:38:41.255 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-18T14:38:41.264 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-18T14:38:41.264 DEBUG:teuthology.orchestra.run.smithi016:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-18T14:38:41.362 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2024-09-18T14:38:41.362 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2024-09-18T14:38:41.362 INFO:teuthology.orchestra.run.smithi016.stdout:*hv01.front.sepi 67.205.162.81 3 u 30 64 377 0.093 2.317 0.485 2024-09-18T14:38:41.363 INFO:teuthology.orchestra.run.smithi016.stdout:+hv02.front.sepi 96.245.170.99 3 u 20 64 377 0.086 4.777 0.298 2024-09-18T14:38:41.363 INFO:teuthology.orchestra.run.smithi016.stdout:+hv03.front.sepi 74.6.168.72 3 u 35 64 377 0.093 1.304 0.263 2024-09-18T14:38:41.363 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2024-09-18T14:38:41.364 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-18T14:38:41.372 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-18T14:38:41.373 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-18T14:38:41.381 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-18T14:38:41.390 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-18T14:38:41.398 INFO:teuthology.task.internal:Duration was 2407.422496 seconds 2024-09-18T14:38:41.399 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-18T14:38:41.407 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-18T14:38:41.407 DEBUG:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-18T14:38:41.527 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-18T14:38:41.527 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi016.front.sepia.ceph.com 2024-09-18T14:38:41.528 DEBUG:teuthology.orchestra.run.smithi016:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-18T14:38:41.542 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-18T14:38:41.543 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-18T14:38:41.587 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-18T14:38:41.588 DEBUG:teuthology.orchestra.run.smithi016:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-18T14:38:41.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-18T14:38:41.777 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-18T14:38:41.777 DEBUG:teuthology.orchestra.run.smithi016:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-18T14:38:41.801 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-18T14:38:41.810 DEBUG:teuthology.orchestra.run.smithi016:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:38:41.874 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = core 2024-09-18T14:38:41.924 DEBUG:teuthology.orchestra.run.smithi016:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-18T14:38:41.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-18T14:38:41.930 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-18T14:38:41.939 INFO:teuthology.task.internal:Transferring archived files... 2024-09-18T14:38:41.940 DEBUG:teuthology.misc:Transferring archived files from smithi016:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-18_09:13:00-rados-wip-bharath11-testing-2024-09-18-1115-quincy-distro-default-smithi/7911080/remote/smithi016 2024-09-18T14:38:41.940 DEBUG:teuthology.orchestra.run.smithi016:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-18T14:38:41.966 INFO:teuthology.task.internal:Removing archive directory... 2024-09-18T14:38:41.966 DEBUG:teuthology.orchestra.run.smithi016:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-18T14:38:42.013 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-18T14:38:42.022 INFO:teuthology.task.internal:Not uploading archives. 2024-09-18T14:38:42.023 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-18T14:38:42.032 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-18T14:38:42.032 DEBUG:teuthology.orchestra.run.smithi016:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-18T14:38:42.039 INFO:teuthology.orchestra.run.smithi016.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 18 14:38 /home/ubuntu/cephtest 2024-09-18T14:38:42.042 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-18T14:38:42.052 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-18T14:38:42.162 INFO:teuthology.run:Summary data: description: rados/singleton/{all/osd-recovery-incomplete mon_election/classic msgr-failures/few msgr/async objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{ubuntu_latest}} duration: 2407.42249584198 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-18T14:38:42.163 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-18T14:38:42.267 INFO:teuthology.run:pass